Oct 31 19:16:44.308296 ip-10-0-142-116 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 31 19:16:44.308307 ip-10-0-142-116 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 31 19:16:44.308314 ip-10-0-142-116 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 31 19:16:44.308538 ip-10-0-142-116 systemd[1]: Failed to start Kubernetes Kubelet. Oct 31 19:16:54.518035 ip-10-0-142-116 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 31 19:16:54.518052 ip-10-0-142-116 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 9b66ad23ebf14ceabb7a991c3f844cd3 -- Oct 31 19:18:43.095220 ip-10-0-142-116 systemd[1]: Starting Kubernetes Kubelet... Oct 31 19:18:43.498313 ip-10-0-142-116 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 19:18:43.498313 ip-10-0-142-116 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 19:18:43.498313 ip-10-0-142-116 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 19:18:43.498313 ip-10-0-142-116 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 19:18:43.504452 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504373 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:18:43.504452 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504450 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:18:43.504452 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504455 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:18:43.504542 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504458 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:18:43.504542 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504461 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:18:43.504542 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504463 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:18:43.504542 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504467 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:18:43.504542 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504470 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:18:43.504542 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504473 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:18:43.504542 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504476 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:18:43.504542 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504479 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:18:43.504542 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504482 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:18:43.504542 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504485 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:18:43.504542 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504487 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:18:43.504542 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504490 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:18:43.504542 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504492 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:18:43.504542 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504495 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:18:43.504542 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504497 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:18:43.504542 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504500 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:18:43.504542 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504503 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:18:43.504542 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504505 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:18:43.504542 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504508 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:18:43.504542 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504510 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:18:43.505023 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504513 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:18:43.505023 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504516 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:18:43.505023 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504518 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:18:43.505023 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504521 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:18:43.505023 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504523 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:18:43.505023 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504526 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:18:43.505023 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504528 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:18:43.505023 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504531 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:18:43.505023 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504533 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:18:43.505023 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504536 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:18:43.505023 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504539 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:18:43.505023 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504542 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:18:43.505023 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504545 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:18:43.505023 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504548 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:18:43.505023 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504550 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:18:43.505023 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504554 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:18:43.505023 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504557 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:18:43.505023 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504559 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:18:43.505023 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504563 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:18:43.505460 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504565 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:18:43.505460 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504568 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:18:43.505460 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504570 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:18:43.505460 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504573 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:18:43.505460 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504576 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:18:43.505460 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504578 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:18:43.505460 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504581 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:18:43.505460 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504583 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:18:43.505460 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504586 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:18:43.505460 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504588 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:18:43.505460 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504591 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:18:43.505460 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504593 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:18:43.505460 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504596 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:18:43.505460 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504599 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:18:43.505460 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504603 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:18:43.505460 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504605 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:18:43.505460 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504608 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:18:43.505460 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504611 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:18:43.505460 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504614 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:18:43.505900 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504617 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:18:43.505900 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504619 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:18:43.505900 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504622 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:18:43.505900 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504624 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:18:43.505900 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504627 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:18:43.505900 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504630 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:18:43.505900 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504633 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:18:43.505900 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504636 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:18:43.505900 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504639 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:18:43.505900 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504642 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:18:43.505900 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504644 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:18:43.505900 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504648 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:18:43.505900 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504650 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:18:43.505900 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504653 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:18:43.505900 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504656 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:18:43.505900 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504658 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:18:43.505900 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504661 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:18:43.505900 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504663 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:18:43.505900 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504666 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:18:43.506353 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504668 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:18:43.506353 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504671 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:18:43.506353 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504674 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:18:43.506353 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504676 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:18:43.506353 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504679 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:18:43.506353 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504681 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:18:43.506353 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504684 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:18:43.506353 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504686 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:18:43.506353 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504689 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:18:43.506353 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504691 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:18:43.506353 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504694 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:18:43.506353 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504697 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:18:43.506353 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504699 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:18:43.506353 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504702 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:18:43.506353 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504704 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:18:43.506353 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504707 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:18:43.506353 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504709 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:18:43.506353 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504712 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:18:43.506353 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504714 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:18:43.506887 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504719 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:18:43.506887 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504724 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:18:43.506887 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504728 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:18:43.506887 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504732 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:18:43.506887 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504736 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:18:43.506887 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504738 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:18:43.506887 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.504742 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:18:43.506887 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505541 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:18:43.506887 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505549 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:18:43.506887 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505552 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:18:43.506887 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505555 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:18:43.506887 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505558 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:18:43.506887 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505561 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:18:43.506887 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505564 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:18:43.506887 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505566 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:18:43.506887 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505569 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:18:43.506887 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505571 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:18:43.506887 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505574 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:18:43.506887 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505577 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:18:43.507605 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505579 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:18:43.507605 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505581 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:18:43.507605 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505584 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:18:43.507605 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505587 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:18:43.507605 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505590 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:18:43.507605 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505592 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:18:43.507605 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505595 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:18:43.507605 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505599 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:18:43.507605 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505603 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:18:43.507605 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505607 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:18:43.507605 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505610 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:18:43.507605 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505613 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:18:43.507605 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505617 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:18:43.507605 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505621 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:18:43.507605 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505623 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:18:43.507605 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505626 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:18:43.507605 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505629 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:18:43.507605 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505631 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:18:43.507605 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505634 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:18:43.508104 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505636 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:18:43.508104 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505640 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:18:43.508104 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505644 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:18:43.508104 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505648 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:18:43.508104 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505655 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:18:43.508104 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505658 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:18:43.508104 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505661 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:18:43.508104 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505664 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:18:43.508104 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505667 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:18:43.508104 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505669 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:18:43.508104 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505672 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:18:43.508104 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505674 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:18:43.508104 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505677 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:18:43.508104 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505679 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:18:43.508104 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505682 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:18:43.508104 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505685 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:18:43.508104 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505687 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:18:43.508104 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505690 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:18:43.508104 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505693 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:18:43.508563 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505696 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:18:43.508563 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505699 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:18:43.508563 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505701 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:18:43.508563 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505704 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:18:43.508563 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505707 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:18:43.508563 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505709 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:18:43.508563 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505712 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:18:43.508563 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505715 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:18:43.508563 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505718 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:18:43.508563 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505721 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:18:43.508563 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505723 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:18:43.508563 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505726 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:18:43.508563 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505729 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:18:43.508563 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505731 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:18:43.508563 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505734 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:18:43.508563 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505736 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:18:43.508563 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505739 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:18:43.508563 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505742 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:18:43.508563 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505745 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:18:43.509032 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505748 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:18:43.509032 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505750 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:18:43.509032 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505753 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:18:43.509032 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505756 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:18:43.509032 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505758 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:18:43.509032 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505761 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:18:43.509032 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505763 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:18:43.509032 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505766 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:18:43.509032 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505768 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:18:43.509032 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505772 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:18:43.509032 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505774 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:18:43.509032 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505777 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:18:43.509032 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505780 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:18:43.509032 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505782 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:18:43.509032 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505785 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:18:43.509032 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505788 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:18:43.509032 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505791 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:18:43.509032 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505793 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:18:43.509032 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505796 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:18:43.509032 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505798 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:18:43.509496 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505801 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:18:43.509496 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505804 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:18:43.509496 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505806 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:18:43.509496 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505809 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:18:43.509496 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505811 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:18:43.509496 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505814 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:18:43.509496 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505816 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:18:43.509496 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505819 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:18:43.509496 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505821 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:18:43.509496 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505824 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:18:43.509496 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505826 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:18:43.509496 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505829 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:18:43.509496 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505832 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:18:43.509496 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505834 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:18:43.509496 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505837 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:18:43.509496 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505839 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:18:43.509496 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.505843 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:18:43.509496 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506330 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:18:43.509496 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506336 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:18:43.509496 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506339 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:18:43.509990 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506341 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:18:43.509990 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506345 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:18:43.509990 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506347 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:18:43.509990 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506350 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:18:43.509990 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506352 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:18:43.509990 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506355 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:18:43.509990 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506358 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:18:43.509990 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506361 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:18:43.509990 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506364 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:18:43.509990 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506367 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:18:43.509990 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506370 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:18:43.509990 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506373 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:18:43.509990 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506375 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:18:43.509990 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506378 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:18:43.509990 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506381 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:18:43.509990 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506384 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:18:43.509990 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506386 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:18:43.509990 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506389 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:18:43.509990 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506391 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:18:43.510433 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506394 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:18:43.510433 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506396 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:18:43.510433 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506399 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:18:43.510433 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506401 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:18:43.510433 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506404 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:18:43.510433 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506407 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:18:43.510433 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506410 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:18:43.510433 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506413 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:18:43.510433 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506415 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:18:43.510433 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506418 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:18:43.510433 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506420 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:18:43.510433 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506423 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:18:43.510433 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506425 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:18:43.510433 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506428 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:18:43.510433 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506431 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:18:43.510433 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506437 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:18:43.510433 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506440 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:18:43.510433 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506442 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:18:43.510433 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506445 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:18:43.510433 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506448 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:18:43.510937 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506450 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:18:43.510937 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506453 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:18:43.510937 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506455 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:18:43.510937 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506458 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:18:43.510937 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506461 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:18:43.510937 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506463 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:18:43.510937 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506467 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:18:43.510937 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506469 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:18:43.510937 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506472 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:18:43.510937 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506474 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:18:43.510937 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506477 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:18:43.510937 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506479 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:18:43.510937 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506481 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:18:43.510937 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506484 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:18:43.510937 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506487 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:18:43.510937 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506489 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:18:43.510937 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506492 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:18:43.510937 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506495 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:18:43.510937 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506499 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:18:43.511384 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506503 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:18:43.511384 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506507 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:18:43.511384 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506510 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:18:43.511384 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506512 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:18:43.511384 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506515 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:18:43.511384 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506518 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:18:43.511384 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506520 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:18:43.511384 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506523 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:18:43.511384 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506527 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:18:43.511384 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506529 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:18:43.511384 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506532 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:18:43.511384 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506535 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:18:43.511384 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506538 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:18:43.511384 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506542 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:18:43.511384 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506546 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:18:43.511384 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506549 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:18:43.511384 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506553 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:18:43.511384 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506555 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:18:43.511384 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506560 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:18:43.511824 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506563 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:18:43.511824 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506566 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:18:43.511824 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506568 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:18:43.511824 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506571 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:18:43.511824 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506573 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:18:43.511824 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506576 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:18:43.511824 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506579 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:18:43.511824 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506581 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:18:43.511824 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506584 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:18:43.511824 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506586 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:18:43.511824 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506589 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:18:43.511824 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506591 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:18:43.511824 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506631 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:18:43.511824 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506684 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:18:43.511824 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506759 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:18:43.511824 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506772 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:18:43.511824 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506778 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:18:43.511824 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506784 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:18:43.511824 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506791 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:18:43.511824 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506796 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:18:43.512351 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506802 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:18:43.512351 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506815 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:18:43.512351 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506820 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:18:43.512351 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506825 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:18:43.512351 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506829 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:18:43.512351 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.506840 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:18:43.512351 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507054 2566 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 19:18:43.512351 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507065 2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 19:18:43.512351 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507077 2566 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 19:18:43.512351 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507085 2566 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 19:18:43.512351 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507092 2566 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 19:18:43.512351 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507105 2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 19:18:43.512351 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507112 2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 19:18:43.512351 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507119 2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 19:18:43.512351 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507124 2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 19:18:43.512351 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507130 2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 19:18:43.512351 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507136 2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 19:18:43.512351 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507142 2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 19:18:43.512351 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507147 2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 19:18:43.512351 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507157 2566 flags.go:64] FLAG: --cgroup-root="" Oct 31 19:18:43.512351 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507162 2566 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 19:18:43.512351 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507167 2566 flags.go:64] FLAG: --client-ca-file="" Oct 31 19:18:43.512351 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507172 2566 flags.go:64] FLAG: --cloud-provider="external" Oct 31 19:18:43.512895 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507177 2566 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 19:18:43.512895 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507185 2566 flags.go:64] FLAG: --cluster-domain="" Oct 31 19:18:43.512895 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507189 2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 19:18:43.512895 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507195 2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 19:18:43.512895 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507202 2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 19:18:43.512895 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507212 2566 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 19:18:43.512895 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507219 2566 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 19:18:43.512895 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507223 2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 19:18:43.512895 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507228 2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 19:18:43.512895 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507233 2566 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 19:18:43.512895 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507239 2566 flags.go:64] FLAG: --contention-profiling="false" Oct 31 19:18:43.512895 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507244 2566 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 19:18:43.512895 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507249 2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 19:18:43.512895 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507259 2566 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 19:18:43.512895 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507264 2566 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 19:18:43.512895 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507271 2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 19:18:43.512895 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507276 2566 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 19:18:43.512895 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507281 2566 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 19:18:43.512895 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507286 2566 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 19:18:43.512895 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507291 2566 flags.go:64] FLAG: --enable-server="true" Oct 31 19:18:43.512895 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507296 2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 19:18:43.512895 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507310 2566 flags.go:64] FLAG: --event-burst="100" Oct 31 19:18:43.512895 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507316 2566 flags.go:64] FLAG: --event-qps="50" Oct 31 19:18:43.512895 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507321 2566 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 19:18:43.513537 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507326 2566 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 19:18:43.513537 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507331 2566 flags.go:64] FLAG: --eviction-hard="" Oct 31 19:18:43.513537 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507339 2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 19:18:43.513537 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507344 2566 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 19:18:43.513537 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507349 2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 19:18:43.513537 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507354 2566 flags.go:64] FLAG: --eviction-soft="" Oct 31 19:18:43.513537 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507364 2566 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 19:18:43.513537 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507369 2566 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 19:18:43.513537 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507374 2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 19:18:43.513537 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507380 2566 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 19:18:43.513537 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507385 2566 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 19:18:43.513537 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507390 2566 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 19:18:43.513537 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507395 2566 flags.go:64] FLAG: --feature-gates="" Oct 31 19:18:43.513537 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507402 2566 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 19:18:43.513537 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507412 2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 19:18:43.513537 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507417 2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 19:18:43.513537 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507423 2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 19:18:43.513537 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507428 2566 flags.go:64] FLAG: --healthz-port="10248" Oct 31 19:18:43.513537 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507432 2566 flags.go:64] FLAG: --help="false" Oct 31 19:18:43.513537 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507437 2566 flags.go:64] FLAG: --hostname-override="ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:18:43.513537 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507443 2566 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 19:18:43.513537 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507449 2566 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 19:18:43.513537 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507454 2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 19:18:43.514077 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507465 2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 19:18:43.514077 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507471 2566 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 19:18:43.514077 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507476 2566 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 19:18:43.514077 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507481 2566 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 19:18:43.514077 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507486 2566 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 19:18:43.514077 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507491 2566 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 19:18:43.514077 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507500 2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 19:18:43.514077 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507505 2566 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 19:18:43.514077 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507515 2566 flags.go:64] FLAG: --kube-reserved="" Oct 31 19:18:43.514077 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507520 2566 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 19:18:43.514077 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507525 2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 19:18:43.514077 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507531 2566 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 19:18:43.514077 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507535 2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 19:18:43.514077 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507540 2566 flags.go:64] FLAG: --lock-file="" Oct 31 19:18:43.514077 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507545 2566 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 19:18:43.514077 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507550 2566 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 19:18:43.514077 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507554 2566 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 19:18:43.514077 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507568 2566 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 19:18:43.514077 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507573 2566 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 19:18:43.514077 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507578 2566 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 19:18:43.514077 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507582 2566 flags.go:64] FLAG: --logging-format="text" Oct 31 19:18:43.514077 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507588 2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 19:18:43.514077 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507594 2566 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 19:18:43.514077 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507598 2566 flags.go:64] FLAG: --manifest-url="" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507602 2566 flags.go:64] FLAG: --manifest-url-header="" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507900 2566 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507925 2566 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507933 2566 flags.go:64] FLAG: --max-pods="110" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507937 2566 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507941 2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507944 2566 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507948 2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507951 2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507954 2566 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507957 2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507967 2566 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507971 2566 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507974 2566 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507977 2566 flags.go:64] FLAG: --pod-cidr="" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507980 2566 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507984 2566 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507987 2566 flags.go:64] FLAG: --pods-per-core="0" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507989 2566 flags.go:64] FLAG: --port="10250" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507992 2566 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507996 2566 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0018c164098861af7" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.507999 2566 flags.go:64] FLAG: --qos-reserved="" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508002 2566 flags.go:64] FLAG: --read-only-port="10255" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508006 2566 flags.go:64] FLAG: --register-node="true" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508009 2566 flags.go:64] FLAG: --register-with-taints="" Oct 31 19:18:43.514679 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508013 2566 flags.go:64] FLAG: --registry-burst="10" Oct 31 19:18:43.515331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508016 2566 flags.go:64] FLAG: --registry-qps="5" Oct 31 19:18:43.515331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508019 2566 flags.go:64] FLAG: --reserved-cpus="" Oct 31 19:18:43.515331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508023 2566 flags.go:64] FLAG: --reserved-memory="" Oct 31 19:18:43.515331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508029 2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 19:18:43.515331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508032 2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 19:18:43.515331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508037 2566 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 19:18:43.515331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508040 2566 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 19:18:43.515331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508043 2566 flags.go:64] FLAG: --runonce="false" Oct 31 19:18:43.515331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508046 2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 19:18:43.515331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508050 2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 19:18:43.515331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508053 2566 flags.go:64] FLAG: --seccomp-default="false" Oct 31 19:18:43.515331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508056 2566 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 19:18:43.515331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508059 2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 19:18:43.515331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508062 2566 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 19:18:43.515331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508065 2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 19:18:43.515331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508068 2566 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 19:18:43.515331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508071 2566 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 19:18:43.515331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508074 2566 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 19:18:43.515331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508078 2566 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 19:18:43.515331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508080 2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 19:18:43.515331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508084 2566 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 19:18:43.515331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508087 2566 flags.go:64] FLAG: --system-cgroups="" Oct 31 19:18:43.515331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508090 2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 19:18:43.515331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508096 2566 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 19:18:43.515331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508099 2566 flags.go:64] FLAG: --tls-cert-file="" Oct 31 19:18:43.515941 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508102 2566 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 19:18:43.515941 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508108 2566 flags.go:64] FLAG: --tls-min-version="" Oct 31 19:18:43.515941 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508111 2566 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 19:18:43.515941 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508114 2566 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 19:18:43.515941 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508117 2566 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 19:18:43.515941 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508121 2566 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 19:18:43.515941 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508124 2566 flags.go:64] FLAG: --v="2" Oct 31 19:18:43.515941 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508128 2566 flags.go:64] FLAG: --version="false" Oct 31 19:18:43.515941 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508133 2566 flags.go:64] FLAG: --vmodule="" Oct 31 19:18:43.515941 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508137 2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 19:18:43.515941 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508141 2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 19:18:43.515941 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508294 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:18:43.515941 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508299 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:18:43.515941 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508302 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:18:43.515941 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508306 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:18:43.515941 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508308 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:18:43.515941 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508311 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:18:43.515941 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508314 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:18:43.515941 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508317 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:18:43.515941 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508320 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:18:43.515941 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508323 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:18:43.515941 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508325 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:18:43.516548 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508328 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:18:43.516548 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508331 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:18:43.516548 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508333 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:18:43.516548 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508336 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:18:43.516548 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508339 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:18:43.516548 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508343 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:18:43.516548 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508346 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:18:43.516548 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508348 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:18:43.516548 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508351 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:18:43.516548 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508354 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:18:43.516548 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508357 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:18:43.516548 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508360 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:18:43.516548 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508362 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:18:43.516548 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508365 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:18:43.516548 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508367 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:18:43.516548 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508370 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:18:43.516548 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508372 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:18:43.516548 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508375 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:18:43.516548 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508378 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:18:43.516548 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508381 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:18:43.517065 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508384 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:18:43.517065 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508387 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:18:43.517065 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508389 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:18:43.517065 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508391 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:18:43.517065 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508394 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:18:43.517065 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508396 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:18:43.517065 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508399 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:18:43.517065 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508402 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:18:43.517065 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508404 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:18:43.517065 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508407 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:18:43.517065 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508409 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:18:43.517065 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508412 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:18:43.517065 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508414 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:18:43.517065 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508417 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:18:43.517065 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508419 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:18:43.517065 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508422 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:18:43.517065 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508424 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:18:43.517065 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508427 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:18:43.517065 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508430 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:18:43.517543 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508432 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:18:43.517543 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508435 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:18:43.517543 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508437 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:18:43.517543 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508440 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:18:43.517543 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508442 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:18:43.517543 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508445 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:18:43.517543 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508447 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:18:43.517543 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508450 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:18:43.517543 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508452 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:18:43.517543 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508456 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:18:43.517543 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508458 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:18:43.517543 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508461 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:18:43.517543 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508463 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:18:43.517543 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508466 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:18:43.517543 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508471 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:18:43.517543 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508474 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:18:43.517543 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508478 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:18:43.517543 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508481 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:18:43.517543 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508484 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:18:43.518016 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508487 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:18:43.518016 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508489 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:18:43.518016 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508492 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:18:43.518016 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508494 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:18:43.518016 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508497 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:18:43.518016 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508499 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:18:43.518016 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508502 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:18:43.518016 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508505 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:18:43.518016 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508507 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:18:43.518016 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508510 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:18:43.518016 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508512 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:18:43.518016 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508515 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:18:43.518016 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508517 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:18:43.518016 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508520 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:18:43.518016 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508522 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:18:43.518016 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508525 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:18:43.518016 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508528 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:18:43.518016 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508532 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:18:43.518016 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508535 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:18:43.518464 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508538 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:18:43.518464 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508541 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:18:43.518464 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508544 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:18:43.518464 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508546 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:18:43.518464 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508549 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:18:43.518464 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508551 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:18:43.518464 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508554 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:18:43.518464 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508557 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:18:43.518464 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508559 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:18:43.518464 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508562 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:18:43.518464 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508565 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:18:43.518464 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508568 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:18:43.518464 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508571 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:18:43.518464 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508574 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:18:43.518464 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508577 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:18:43.518464 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508579 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:18:43.518464 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508582 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:18:43.518464 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508584 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:18:43.518879 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.508613 2566 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 19:18:43.518879 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508653 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:18:43.518879 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508658 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:18:43.518879 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508661 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:18:43.518879 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508664 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:18:43.518879 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508666 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:18:43.518879 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508669 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:18:43.518879 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508672 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:18:43.518879 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508674 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:18:43.518879 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508677 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:18:43.518879 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508679 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:18:43.518879 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508684 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:18:43.518879 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508688 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:18:43.518879 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508691 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:18:43.518879 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508694 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:18:43.519258 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508696 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:18:43.519258 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508699 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:18:43.519258 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508701 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:18:43.519258 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508704 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:18:43.519258 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508707 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:18:43.519258 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508709 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:18:43.519258 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508712 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:18:43.519258 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508715 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:18:43.519258 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508717 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:18:43.519258 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508720 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:18:43.519258 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508725 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:18:43.519258 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508728 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:18:43.519258 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508730 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:18:43.519258 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508733 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:18:43.519258 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508736 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:18:43.519258 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508739 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:18:43.519258 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508741 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:18:43.519258 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508744 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:18:43.519258 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508747 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:18:43.519692 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508751 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:18:43.519692 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508754 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:18:43.519692 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508756 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:18:43.519692 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508759 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:18:43.519692 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508761 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:18:43.519692 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508764 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:18:43.519692 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508767 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:18:43.519692 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508769 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:18:43.519692 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508771 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:18:43.519692 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508774 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:18:43.519692 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508778 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:18:43.519692 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508781 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:18:43.519692 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508784 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:18:43.519692 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508786 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:18:43.519692 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508789 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:18:43.519692 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508791 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:18:43.519692 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508794 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:18:43.519692 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508797 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:18:43.519692 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508801 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:18:43.519692 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508804 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:18:43.520184 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508808 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:18:43.520184 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508811 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:18:43.520184 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508813 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:18:43.520184 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508818 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:18:43.520184 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508820 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:18:43.520184 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508823 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:18:43.520184 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508825 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:18:43.520184 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508828 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:18:43.520184 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508831 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:18:43.520184 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508834 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:18:43.520184 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508836 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:18:43.520184 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508838 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:18:43.520184 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508841 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:18:43.520184 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508844 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:18:43.520184 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508847 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:18:43.520184 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508849 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:18:43.520184 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508852 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:18:43.520184 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508855 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:18:43.520184 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508858 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:18:43.520635 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508861 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:18:43.520635 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508863 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:18:43.520635 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508866 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:18:43.520635 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508868 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:18:43.520635 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508871 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:18:43.520635 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508874 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:18:43.520635 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508877 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:18:43.520635 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508879 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:18:43.520635 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508882 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:18:43.520635 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508884 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:18:43.520635 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508887 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:18:43.520635 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508891 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:18:43.520635 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508894 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:18:43.520635 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508896 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:18:43.520635 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508899 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:18:43.520635 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508901 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:18:43.520635 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508905 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:18:43.520635 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508924 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:18:43.520635 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508929 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:18:43.520635 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508931 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:18:43.521244 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508934 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:18:43.521244 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508937 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:18:43.521244 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508940 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:18:43.521244 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508942 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:18:43.521244 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508945 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:18:43.521244 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508948 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:18:43.521244 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508950 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:18:43.521244 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508953 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:18:43.521244 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508955 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:18:43.521244 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508958 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:18:43.521244 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508960 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:18:43.521244 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508963 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:18:43.521244 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508966 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:18:43.521244 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.508968 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:18:43.521244 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.515965 2566 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 19:18:43.521244 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.515983 2566 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 19:18:43.521244 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.516941 2566 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 19:18:43.521244 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.516948 2566 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 19:18:43.521244 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517005 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:18:43.521688 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517011 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:18:43.521688 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517014 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:18:43.521688 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517017 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:18:43.521688 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517020 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:18:43.521688 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517023 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:18:43.521688 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517026 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:18:43.521688 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517029 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:18:43.521688 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517032 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:18:43.521688 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517035 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:18:43.521688 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517038 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:18:43.521688 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517041 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:18:43.521688 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517043 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:18:43.521688 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517046 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:18:43.521688 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517049 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:18:43.521688 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517052 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:18:43.521688 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517055 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:18:43.521688 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517057 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:18:43.521688 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517060 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:18:43.521688 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517063 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:18:43.522156 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517065 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:18:43.522156 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517068 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:18:43.522156 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517070 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:18:43.522156 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517073 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:18:43.522156 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517077 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:18:43.522156 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517081 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:18:43.522156 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517084 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:18:43.522156 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517087 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:18:43.522156 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517090 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:18:43.522156 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517094 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:18:43.522156 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517097 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:18:43.522156 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517100 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:18:43.522156 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517102 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:18:43.522156 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517105 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:18:43.522156 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517107 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:18:43.522156 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517110 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:18:43.522156 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517113 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:18:43.522156 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517116 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:18:43.522156 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517119 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:18:43.522633 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517122 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:18:43.522633 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517124 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:18:43.522633 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517127 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:18:43.522633 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517129 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:18:43.522633 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517132 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:18:43.522633 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517135 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:18:43.522633 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517137 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:18:43.522633 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517140 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:18:43.522633 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517143 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:18:43.522633 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517146 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:18:43.522633 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517148 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:18:43.522633 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517151 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:18:43.522633 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517154 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:18:43.522633 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517156 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:18:43.522633 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517159 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:18:43.522633 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517162 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:18:43.522633 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517165 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:18:43.522633 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517167 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:18:43.522633 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517170 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:18:43.522633 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517173 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:18:43.523112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517176 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:18:43.523112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517178 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:18:43.523112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517182 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:18:43.523112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517185 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:18:43.523112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517188 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:18:43.523112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517191 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:18:43.523112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517193 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:18:43.523112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517196 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:18:43.523112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517198 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:18:43.523112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517201 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:18:43.523112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517204 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:18:43.523112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517206 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:18:43.523112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517209 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:18:43.523112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517211 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:18:43.523112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517214 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:18:43.523112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517217 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:18:43.523112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517219 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:18:43.523112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517222 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:18:43.523112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517225 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:18:43.523112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517227 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:18:43.523573 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517230 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:18:43.523573 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517232 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:18:43.523573 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517235 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:18:43.523573 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517237 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:18:43.523573 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517240 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:18:43.523573 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517243 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:18:43.523573 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517246 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:18:43.523573 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517249 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:18:43.523573 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517251 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:18:43.523573 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517254 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:18:43.523573 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517257 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:18:43.523573 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517260 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:18:43.523573 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517262 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:18:43.523573 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517265 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:18:43.523573 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517267 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:18:43.523573 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517270 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:18:43.523573 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517273 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:18:43.523573 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517276 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:18:43.523573 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517279 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:18:43.523573 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517282 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:18:43.524063 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517284 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:18:43.524063 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517287 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:18:43.524063 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517289 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:18:43.524063 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517292 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:18:43.524063 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517294 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:18:43.524063 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517298 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:18:43.524063 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517302 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:18:43.524063 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.517329 2566 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 19:18:43.524063 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517520 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:18:43.524063 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517526 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:18:43.524063 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517529 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:18:43.524063 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517532 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:18:43.524063 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517535 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:18:43.524063 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517538 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:18:43.524063 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517540 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:18:43.524063 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517543 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:18:43.524487 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517546 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:18:43.524487 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517550 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:18:43.524487 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517552 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:18:43.524487 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517556 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:18:43.524487 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517558 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:18:43.524487 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517561 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:18:43.524487 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517564 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:18:43.524487 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517567 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:18:43.524487 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517570 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:18:43.524487 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517573 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:18:43.524487 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517575 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:18:43.524487 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517578 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:18:43.524487 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517581 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:18:43.524487 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517585 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:18:43.524487 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517588 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:18:43.524487 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517590 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:18:43.524487 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517593 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:18:43.524487 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517596 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:18:43.524487 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517598 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:18:43.524487 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517601 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:18:43.524487 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517603 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:18:43.525112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517606 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:18:43.525112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517608 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:18:43.525112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517611 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:18:43.525112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517613 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:18:43.525112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517616 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:18:43.525112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517619 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:18:43.525112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517621 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:18:43.525112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517624 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:18:43.525112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517626 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:18:43.525112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517629 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:18:43.525112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517632 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:18:43.525112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517635 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:18:43.525112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517637 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:18:43.525112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517640 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:18:43.525112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517645 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:18:43.525112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517649 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:18:43.525112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517652 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:18:43.525112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517654 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:18:43.525112 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517658 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:18:43.525552 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517660 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:18:43.525552 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517663 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:18:43.525552 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517666 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:18:43.525552 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517668 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:18:43.525552 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517671 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:18:43.525552 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517673 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:18:43.525552 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517677 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:18:43.525552 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517679 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:18:43.525552 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517682 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:18:43.525552 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517684 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:18:43.525552 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517687 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:18:43.525552 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517689 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:18:43.525552 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517692 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:18:43.525552 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517695 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:18:43.525552 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517697 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:18:43.525552 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517700 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:18:43.525552 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517703 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:18:43.525552 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517707 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:18:43.525552 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517709 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:18:43.526007 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517712 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:18:43.526007 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517716 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:18:43.526007 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517719 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:18:43.526007 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517722 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:18:43.526007 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517725 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:18:43.526007 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517728 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:18:43.526007 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517730 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:18:43.526007 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517733 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:18:43.526007 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517736 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:18:43.526007 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517738 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:18:43.526007 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517741 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:18:43.526007 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517743 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:18:43.526007 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517746 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:18:43.526007 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517749 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:18:43.526007 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517751 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:18:43.526007 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517754 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:18:43.526007 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517757 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:18:43.526007 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517759 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:18:43.526007 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517762 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:18:43.526482 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517765 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:18:43.526482 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517768 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:18:43.526482 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517771 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:18:43.526482 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517773 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:18:43.526482 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517776 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:18:43.526482 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517778 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:18:43.526482 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517781 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:18:43.526482 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517783 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:18:43.526482 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517786 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:18:43.526482 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517789 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:18:43.526482 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517791 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:18:43.526482 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517794 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:18:43.526482 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517797 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:18:43.526482 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517799 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:18:43.526482 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517802 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:18:43.526482 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517805 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:18:43.526482 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517808 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:18:43.526482 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517811 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:18:43.526482 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517813 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:18:43.526482 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517816 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:18:43.526952 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.517839 2566 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 19:18:43.526952 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517884 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:18:43.526952 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517888 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:18:43.526952 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517892 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:18:43.526952 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517895 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:18:43.526952 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517897 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:18:43.526952 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517900 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:18:43.526952 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517903 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:18:43.526952 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517906 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:18:43.526952 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517923 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:18:43.526952 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517928 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:18:43.526952 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517932 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:18:43.526952 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517935 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:18:43.526952 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517938 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:18:43.526952 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517954 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:18:43.527293 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517958 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:18:43.527293 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517961 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:18:43.527293 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517964 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:18:43.527293 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517966 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:18:43.527293 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517970 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:18:43.527293 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517972 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:18:43.527293 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517975 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:18:43.527293 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517978 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:18:43.527293 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517980 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:18:43.527293 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517983 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:18:43.527293 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517985 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:18:43.527293 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517988 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:18:43.527293 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517990 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:18:43.527293 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517993 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:18:43.527293 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517996 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:18:43.527293 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.517999 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:18:43.527293 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518002 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:18:43.527293 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518006 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:18:43.527293 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518008 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:18:43.527734 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518011 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:18:43.527734 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518014 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:18:43.527734 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518016 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:18:43.527734 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518019 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:18:43.527734 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518022 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:18:43.527734 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518025 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:18:43.527734 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518028 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:18:43.527734 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518031 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:18:43.527734 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518033 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:18:43.527734 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518036 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:18:43.527734 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518038 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:18:43.527734 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518041 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:18:43.527734 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518044 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:18:43.527734 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518047 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:18:43.527734 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518049 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:18:43.527734 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518052 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:18:43.527734 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518055 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:18:43.527734 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518057 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:18:43.527734 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518060 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:18:43.528196 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518062 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:18:43.528196 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518065 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:18:43.528196 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518067 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:18:43.528196 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518070 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:18:43.528196 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518073 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:18:43.528196 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518075 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:18:43.528196 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518078 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:18:43.528196 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518081 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:18:43.528196 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518083 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:18:43.528196 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518086 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:18:43.528196 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518089 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:18:43.528196 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518091 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:18:43.528196 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518094 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:18:43.528196 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518097 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:18:43.528196 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518099 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:18:43.528196 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518102 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:18:43.528196 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518104 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:18:43.528196 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518107 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:18:43.528196 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518109 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:18:43.528196 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518112 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:18:43.528652 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518114 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:18:43.528652 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518117 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:18:43.528652 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518119 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:18:43.528652 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518122 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:18:43.528652 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518124 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:18:43.528652 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518127 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:18:43.528652 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518129 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:18:43.528652 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518134 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:18:43.528652 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518139 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:18:43.528652 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518142 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:18:43.528652 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518144 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:18:43.528652 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518147 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:18:43.528652 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518150 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:18:43.528652 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518152 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:18:43.528652 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518155 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:18:43.528652 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518157 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:18:43.528652 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518160 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:18:43.528652 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518162 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:18:43.528652 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518165 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:18:43.528652 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518168 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:18:43.529229 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518171 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:18:43.529229 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518174 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:18:43.529229 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518176 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:18:43.529229 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518179 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:18:43.529229 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518181 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:18:43.529229 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518184 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:18:43.529229 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518186 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:18:43.529229 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518189 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:18:43.529229 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518191 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:18:43.529229 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518194 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:18:43.529229 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518197 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:18:43.529229 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518199 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:18:43.529229 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518202 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:18:43.529229 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:43.518204 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:18:43.529229 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.518286 2566 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 19:18:43.529229 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.520762 2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 19:18:43.529229 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.521797 2566 server.go:1014] "Starting client certificate rotation" Oct 31 19:18:43.529229 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.521890 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 19:18:43.529229 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.521934 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 19:18:43.529671 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.524188 2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 19:18:43.529671 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.525494 2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 19:18:43.537198 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.537182 2566 log.go:25] "Validated CRI v1 runtime API" Oct 31 19:18:43.541816 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.541798 2566 log.go:25] "Validated CRI v1 image API" Oct 31 19:18:43.542695 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.542670 2566 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 19:18:43.547789 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.547761 2566 fs.go:135] Filesystem UUIDs: map[1a714163-8081-4292-a582-ab03cd37d2fc:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 91ac38ec-3f9c-4ad1-8eff-822e29738eb1:/dev/nvme0n1p3] Oct 31 19:18:43.547847 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.547788 2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 19:18:43.554484 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.554372 2566 manager.go:217] Machine: {Timestamp:2026-10-31 19:18:43.551945753 +0000 UTC m=+0.366039975 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105501 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2512f715f416bb51e08e0bfb77805a SystemUUID:ec2512f7-15f4-16bb-51e0-8e0bfb77805a BootID:9b66ad23-ebf1-4cea-bb7a-991c3f844cd3 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:db:82:d7:7d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:db:82:d7:7d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:03:3c:e6:c4:55 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 19:18:43.554484 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.554483 2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 19:18:43.554575 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.554562 2566 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 19:18:43.558248 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.558219 2566 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 19:18:43.558387 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.558250 2566 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-116.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 19:18:43.558435 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.558389 2566 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 19:18:43.558435 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.558397 2566 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 19:18:43.558490 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.558445 2566 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 19:18:43.558490 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.558463 2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 19:18:43.559779 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.559763 2566 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 19:18:43.564453 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.564440 2566 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 19:18:43.564614 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.564605 2566 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 19:18:43.565453 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.565444 2566 kubelet.go:493] "Attempting to sync node with API server" Oct 31 19:18:43.565485 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.565458 2566 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 19:18:43.565485 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.565468 2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 19:18:43.565485 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.565477 2566 kubelet.go:405] "Adding apiserver pod source" Oct 31 19:18:43.565485 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.565485 2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 19:18:43.565896 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.565884 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 19:18:43.568438 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.568414 2566 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 19:18:43.576066 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.576053 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 19:18:43.577037 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.577025 2566 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 19:18:43.577074 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.577055 2566 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 19:18:43.577273 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.577263 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 19:18:43.577314 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.577279 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 19:18:43.577314 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.577286 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 19:18:43.577314 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.577292 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 19:18:43.577314 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.577298 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 19:18:43.577314 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.577309 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 19:18:43.577995 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.577987 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 19:18:43.578025 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.577997 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 19:18:43.578058 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.578029 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 19:18:43.578058 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.578038 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 19:18:43.578058 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.578054 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 19:18:43.578151 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.578062 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 19:18:43.579392 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.579384 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 19:18:43.579428 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.579393 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 19:18:43.583394 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.583381 2566 server.go:1263] "Started kubelet" Oct 31 19:18:43.583481 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.583463 2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-116.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 19:18:43.583581 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.583525 2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 19:18:43.583683 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.583595 2566 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 19:18:43.583740 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.583694 2566 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 19:18:43.584241 ip-10-0-142-116 systemd[1]: Started Kubernetes Kubelet. Oct 31 19:18:43.585105 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.585087 2566 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 19:18:43.585828 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.585813 2566 server.go:317] "Adding debug handlers to kubelet server" Oct 31 19:18:43.589299 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.588962 2566 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 19:18:43.590129 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:43.590107 2566 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 19:18:43.594279 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.594263 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 19:18:43.595644 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.595627 2566 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 19:18:43.596112 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:43.596089 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-116.us-east-2.compute.internal\" not found" Oct 31 19:18:43.596699 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.596678 2566 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 19:18:43.596699 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.596688 2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 19:18:43.596825 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.596710 2566 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 19:18:43.596869 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.596826 2566 reconstruct.go:97] "Volume reconstruction finished" Oct 31 19:18:43.596869 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.596837 2566 reconciler.go:29] "Reconciler: start to sync state" Oct 31 19:18:43.596983 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.596889 2566 factory.go:55] Registering systemd factory Oct 31 19:18:43.596983 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.596904 2566 factory.go:223] Registration of the systemd container factory successfully Oct 31 19:18:43.597122 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.597108 2566 factory.go:153] Registering CRI-O factory Oct 31 19:18:43.597159 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.597126 2566 factory.go:223] Registration of the crio container factory successfully Oct 31 19:18:43.597190 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.597172 2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 19:18:43.597221 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.597189 2566 factory.go:103] Registering Raw factory Oct 31 19:18:43.597221 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.597203 2566 manager.go:1199] Started watching for new ooms in manager Oct 31 19:18:43.598225 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.598210 2566 manager.go:319] Starting recovery of all containers Oct 31 19:18:43.606428 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.606407 2566 manager.go:324] Recovery completed Oct 31 19:18:43.608821 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:43.608789 2566 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 19:18:43.609809 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:43.609719 2566 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-116.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 19:18:43.609809 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:43.609784 2566 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-116.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 19:18:43.611424 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:43.610105 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-116.us-east-2.compute.internal.18e3b36522f84012 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-116.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-116.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 19:18:43.58335285 +0000 UTC m=+0.397447072,LastTimestamp:2026-10-31 19:18:43.58335285 +0000 UTC m=+0.397447072,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-116.us-east-2.compute.internal,}" Oct 31 19:18:43.612096 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.612082 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:18:43.613751 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.613728 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:18:43.613823 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.613766 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:18:43.613823 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.613781 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:18:43.614367 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.614355 2566 cpu_manager.go:226] "Starting" policy="none" Oct 31 19:18:43.614367 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.614363 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:18:43.614481 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.614366 2566 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 19:18:43.614481 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.614416 2566 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 19:18:43.615160 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.615144 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:18:43.615230 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.615171 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:18:43.615230 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.615185 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:18:43.616688 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.616673 2566 policy_none.go:50] "Start" Oct 31 19:18:43.616756 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.616691 2566 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 19:18:43.616756 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.616705 2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 19:18:43.619045 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:43.619017 2566 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 19:18:43.619045 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:43.618950 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-116.us-east-2.compute.internal.18e3b36524c80b76 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-116.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-116.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-116.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 19:18:43.613748086 +0000 UTC m=+0.427842314,LastTimestamp:2026-10-31 19:18:43.613748086 +0000 UTC m=+0.427842314,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-116.us-east-2.compute.internal,}" Oct 31 19:18:43.619194 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.619117 2566 policy_none.go:44] "Start" Oct 31 19:18:43.623900 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:43.623839 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-116.us-east-2.compute.internal.18e3b36524c871ac default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-116.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-116.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-116.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 19:18:43.613774252 +0000 UTC m=+0.427868480,LastTimestamp:2026-10-31 19:18:43.613774252 +0000 UTC m=+0.427868480,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-116.us-east-2.compute.internal,}" Oct 31 19:18:43.630334 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:43.630280 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-116.us-east-2.compute.internal.18e3b36524c89b7c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-116.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-116.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-116.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 19:18:43.613784956 +0000 UTC m=+0.427879178,LastTimestamp:2026-10-31 19:18:43.613784956 +0000 UTC m=+0.427879178,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-116.us-east-2.compute.internal,}" Oct 31 19:18:43.640115 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:43.640035 2566 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-116.us-east-2.compute.internal.18e3b36524c80b76\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-116.us-east-2.compute.internal.18e3b36524c80b76 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-116.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-116.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-116.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 19:18:43.613748086 +0000 UTC m=+0.427842314,LastTimestamp:2026-10-31 19:18:43.615158182 +0000 UTC m=+0.429252407,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-116.us-east-2.compute.internal,}" Oct 31 19:18:43.652777 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:43.652718 2566 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-116.us-east-2.compute.internal.18e3b36524c871ac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-116.us-east-2.compute.internal.18e3b36524c871ac default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-116.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-116.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-116.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 19:18:43.613774252 +0000 UTC m=+0.427868480,LastTimestamp:2026-10-31 19:18:43.615177598 +0000 UTC m=+0.429271821,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-116.us-east-2.compute.internal,}" Oct 31 19:18:43.654488 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.654459 2566 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 19:18:43.656847 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.656835 2566 manager.go:341] "Starting Device Plugin manager" Oct 31 19:18:43.656980 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:43.656863 2566 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 19:18:43.656980 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.656871 2566 server.go:87] "Starting device plugin registration server" Oct 31 19:18:43.657102 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.657090 2566 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 19:18:43.657145 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.657102 2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 19:18:43.657195 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.657179 2566 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 19:18:43.657270 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.657253 2566 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 19:18:43.657270 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.657267 2566 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 19:18:43.657714 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:43.657694 2566 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 19:18:43.657778 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:43.657739 2566 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-116.us-east-2.compute.internal\" not found" Oct 31 19:18:43.667275 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:43.667210 2566 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-116.us-east-2.compute.internal.18e3b36524c89b7c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-116.us-east-2.compute.internal.18e3b36524c89b7c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-116.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-116.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-116.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 19:18:43.613784956 +0000 UTC m=+0.427879178,LastTimestamp:2026-10-31 19:18:43.615189166 +0000 UTC m=+0.429283388,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-116.us-east-2.compute.internal,}" Oct 31 19:18:43.693373 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:43.681634 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-116.us-east-2.compute.internal.18e3b365275919ac default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-116.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated limits on system reserved cgroup /system.slice,Source:EventSource{Component:kubelet,Host:ip-10-0-142-116.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 19:18:43.656808876 +0000 UTC m=+0.470903092,LastTimestamp:2026-10-31 19:18:43.656808876 +0000 UTC m=+0.470903092,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-116.us-east-2.compute.internal,}" Oct 31 19:18:43.693373 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:43.691965 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-116.us-east-2.compute.internal.18e3b3652786ddad default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-116.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-142-116.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 19:18:43.659808173 +0000 UTC m=+0.473902402,LastTimestamp:2026-10-31 19:18:43.659808173 +0000 UTC m=+0.473902402,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-116.us-east-2.compute.internal,}" Oct 31 19:18:43.721700 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.721676 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6q2l2" Oct 31 19:18:43.729276 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.729248 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 19:18:43.730433 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.730420 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 19:18:43.730500 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.730441 2566 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 19:18:43.730500 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.730463 2566 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 19:18:43.730590 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:43.730502 2566 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 19:18:43.738401 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.738379 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6q2l2" Oct 31 19:18:43.751038 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.750991 2566 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:18:43.758017 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.758005 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:18:43.758889 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.758871 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:18:43.758975 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.758924 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:18:43.758975 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.758941 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:18:43.758975 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.758969 2566 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:18:43.771112 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.771091 2566 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:18:43.771209 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:43.771117 2566 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-116.us-east-2.compute.internal\": node \"ip-10-0-142-116.us-east-2.compute.internal\" not found" Oct 31 19:18:43.830728 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.830563 2566 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal"] Oct 31 19:18:43.830885 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.830759 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:18:43.831681 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.831666 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:18:43.831753 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.831693 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:18:43.831753 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.831703 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:18:43.834086 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.834073 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:18:43.834246 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.834232 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:18:43.834299 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.834267 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:18:43.834799 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.834786 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:18:43.834852 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.834811 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:18:43.834852 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.834821 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:18:43.834929 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.834792 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:18:43.834929 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.834876 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:18:43.834929 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.834886 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:18:43.837655 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.837638 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:18:43.837749 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.837664 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:18:43.838372 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.838357 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:18:43.838455 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.838387 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:18:43.838455 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.838402 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:18:43.856405 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:43.856391 2566 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-116.us-east-2.compute.internal\" not found" node="ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:18:43.858518 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:43.858503 2566 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-116.us-east-2.compute.internal\" not found" node="ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:18:43.898015 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.897991 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2993240a68b51a645ff6b733bc57689d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal\" (UID: \"2993240a68b51a645ff6b733bc57689d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:18:43.898109 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.898020 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2993240a68b51a645ff6b733bc57689d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal\" (UID: \"2993240a68b51a645ff6b733bc57689d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:18:43.898109 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.898039 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/956815bb47502a7ffa36d27c2bef7876-config\") pod \"kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal\" (UID: \"956815bb47502a7ffa36d27c2bef7876\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:18:43.938880 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:43.938855 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-116.us-east-2.compute.internal\" not found" Oct 31 19:18:43.999073 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.999045 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/956815bb47502a7ffa36d27c2bef7876-config\") pod \"kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal\" (UID: \"956815bb47502a7ffa36d27c2bef7876\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:18:43.999173 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.999077 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2993240a68b51a645ff6b733bc57689d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal\" (UID: \"2993240a68b51a645ff6b733bc57689d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:18:43.999173 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.999095 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2993240a68b51a645ff6b733bc57689d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal\" (UID: \"2993240a68b51a645ff6b733bc57689d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:18:43.999173 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.999151 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2993240a68b51a645ff6b733bc57689d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal\" (UID: \"2993240a68b51a645ff6b733bc57689d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:18:43.999173 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.999158 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/956815bb47502a7ffa36d27c2bef7876-config\") pod \"kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal\" (UID: \"956815bb47502a7ffa36d27c2bef7876\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:18:43.999307 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:43.999157 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2993240a68b51a645ff6b733bc57689d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal\" (UID: \"2993240a68b51a645ff6b733bc57689d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:18:44.039063 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:44.039004 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-116.us-east-2.compute.internal\" not found" Oct 31 19:18:44.139757 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:44.139715 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-116.us-east-2.compute.internal\" not found" Oct 31 19:18:44.159916 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:44.159883 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:18:44.162049 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:44.162033 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:18:44.240758 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:44.240728 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-116.us-east-2.compute.internal\" not found" Oct 31 19:18:44.341755 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:44.341712 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-116.us-east-2.compute.internal\" not found" Oct 31 19:18:44.444195 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:44.444161 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-116.us-east-2.compute.internal\" not found" Oct 31 19:18:44.503531 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:44.503502 2566 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:18:44.523052 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:44.523024 2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 19:18:44.523168 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:44.523148 2566 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 19:18:44.523211 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:44.523189 2566 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 19:18:44.545949 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:44.545926 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-116.us-east-2.compute.internal\" not found" Oct 31 19:18:44.599373 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:44.599269 2566 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:18:44.646966 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:44.646930 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-116.us-east-2.compute.internal\" not found" Oct 31 19:18:44.742994 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:44.742955 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 19:13:43 +0000 UTC" deadline="2028-07-23 23:10:25.404759864 +0000 UTC" Oct 31 19:18:44.743131 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:44.743003 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15147h51m40.661760948s" Oct 31 19:18:44.747065 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:44.747044 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-116.us-east-2.compute.internal\" not found" Oct 31 19:18:44.779237 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:44.779216 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:18:44.780858 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:44.780842 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:18:44.780978 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:44.780873 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:18:44.780978 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:44.780885 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:18:44.847194 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:44.847163 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-116.us-east-2.compute.internal\" not found" Oct 31 19:18:44.891126 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:44.891073 2566 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:18:44.896536 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:44.896518 2566 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:18:44.907339 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:44.907307 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2993240a68b51a645ff6b733bc57689d.slice/crio-a266a13bd4e51c27d828d0bb7e1adbb395a4f2faed8940c3282bc8fa68722f9f WatchSource:0}: Error finding container a266a13bd4e51c27d828d0bb7e1adbb395a4f2faed8940c3282bc8fa68722f9f: Status 404 returned error can't find the container with id a266a13bd4e51c27d828d0bb7e1adbb395a4f2faed8940c3282bc8fa68722f9f Oct 31 19:18:44.915193 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:44.915173 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 19:18:44.918648 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:44.918611 2566 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 19:18:44.919138 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:44.919114 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod956815bb47502a7ffa36d27c2bef7876.slice/crio-b1940201e2acbb0579e67e09eba8ea212ddb141274f55d9b7d9da554b57e149f WatchSource:0}: Error finding container b1940201e2acbb0579e67e09eba8ea212ddb141274f55d9b7d9da554b57e149f: Status 404 returned error can't find the container with id b1940201e2acbb0579e67e09eba8ea212ddb141274f55d9b7d9da554b57e149f Oct 31 19:18:44.920221 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:44.920200 2566 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:18:44.931610 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:44.931594 2566 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 19:18:45.566648 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.566609 2566 apiserver.go:52] "Watching apiserver" Oct 31 19:18:45.590691 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.590631 2566 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 19:18:45.591134 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.591103 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-wxdtf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7","openshift-image-registry/node-ca-8m482","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-4ngtp","openshift-dns/node-resolver-85mq2","kube-system/global-pull-secret-syncer-lhtkb","openshift-multus/multus-additional-cni-plugins-bhqht","openshift-multus/network-metrics-daemon-dfbrp","openshift-ovn-kubernetes/ovnkube-node-m2pjd","kube-system/kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal","openshift-multus/multus-tm5fh","openshift-network-diagnostics/network-check-target-5mkxz","openshift-network-operator/iptables-alerter-qn24x"] Oct 31 19:18:45.594063 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.594041 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8m482" Oct 31 19:18:45.596108 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.596084 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 19:18:45.597575 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.597553 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.601298 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.601275 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.601780 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.601762 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 19:18:45.601916 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.601902 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 19:18:45.601968 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.601935 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 19:18:45.602126 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.602101 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 19:18:45.602168 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.602124 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gfbm\"" Oct 31 19:18:45.602591 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.602572 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.603390 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.603351 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-w5n5s\"" Oct 31 19:18:45.605336 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.605315 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-85mq2" Oct 31 19:18:45.606188 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.605960 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 19:18:45.609117 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.609095 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:18:45.609213 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:45.609186 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtkb" podUID="9255a1d5-2514-441e-b942-6c39843e38f8" Oct 31 19:18:45.609318 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.609300 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-system-cni-dir\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.609390 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.609331 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-multus-cni-dir\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.609390 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.609347 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5708be73-8282-4caa-bce3-46d4b708860e-cni-binary-copy\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.609390 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.609373 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-var-lib-cni-bin\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.609549 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.609431 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-var-lib-cni-multus\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.609549 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.609480 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-multus-conf-dir\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.609549 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.609531 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-run-k8s-cni-cncf-io\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.609850 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.609567 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-run-netns\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.609850 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.609609 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-var-lib-kubelet\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.609850 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.609632 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-hostroot\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.609850 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.609658 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5708be73-8282-4caa-bce3-46d4b708860e-multus-daemon-config\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.609850 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.609697 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-run-multus-certs\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.609850 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.609724 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-etc-kubernetes\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.609850 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.609768 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/84f5548f-7d70-4391-89c9-3bd3c1784717-serviceca\") pod \"node-ca-8m482\" (UID: \"84f5548f-7d70-4391-89c9-3bd3c1784717\") " pod="openshift-image-registry/node-ca-8m482" Oct 31 19:18:45.609850 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.609818 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-cnibin\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.610292 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.609856 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mjmg\" (UniqueName: \"kubernetes.io/projected/5708be73-8282-4caa-bce3-46d4b708860e-kube-api-access-9mjmg\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.610292 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.609881 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/84f5548f-7d70-4391-89c9-3bd3c1784717-host\") pod \"node-ca-8m482\" (UID: \"84f5548f-7d70-4391-89c9-3bd3c1784717\") " pod="openshift-image-registry/node-ca-8m482" Oct 31 19:18:45.610292 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.609906 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msgz5\" (UniqueName: \"kubernetes.io/projected/84f5548f-7d70-4391-89c9-3bd3c1784717-kube-api-access-msgz5\") pod \"node-ca-8m482\" (UID: \"84f5548f-7d70-4391-89c9-3bd3c1784717\") " pod="openshift-image-registry/node-ca-8m482" Oct 31 19:18:45.610292 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.609981 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-multus-socket-dir-parent\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.612592 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.612570 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:18:45.615518 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.615500 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:18:45.615614 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:45.615553 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfbrp" podUID="88a1fa4b-1191-434a-8e7c-783163ece24d" Oct 31 19:18:45.616791 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.616762 2566 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 19:18:45.618180 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.618157 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wxdtf" Oct 31 19:18:45.620852 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.620832 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:18:45.625371 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.623972 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:18:45.625371 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:45.624083 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5mkxz" podUID="23a10416-51b1-415f-b2fc-892b4d662b1f" Oct 31 19:18:45.629091 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.629070 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qn24x" Oct 31 19:18:45.648288 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.648266 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2jvbm" Oct 31 19:18:45.663226 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.663201 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2jvbm" Oct 31 19:18:45.698222 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.698202 2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 19:18:45.709476 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.709453 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rs77w\"" Oct 31 19:18:45.709589 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.709572 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 19:18:45.709651 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.709632 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 19:18:45.710177 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710158 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 19:18:45.710264 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710236 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 19:18:45.710397 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710379 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-multus-socket-dir-parent\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.710452 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710417 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-run-ovn\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.710452 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710446 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-cni-netd\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.710517 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710471 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9255a1d5-2514-441e-b942-6c39843e38f8-original-pull-secret\") pod \"global-pull-secret-syncer-lhtkb\" (UID: \"9255a1d5-2514-441e-b942-6c39843e38f8\") " pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:18:45.710517 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710500 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7djgh\" (UniqueName: \"kubernetes.io/projected/23a10416-51b1-415f-b2fc-892b4d662b1f-kube-api-access-7djgh\") pod \"network-check-target-5mkxz\" (UID: \"23a10416-51b1-415f-b2fc-892b4d662b1f\") " pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:18:45.710601 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710527 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-multus-cni-dir\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.710601 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710552 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-var-lib-cni-bin\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.710601 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710577 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-sysctl-d\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.710601 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710597 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9255a1d5-2514-441e-b942-6c39843e38f8-kubelet-config\") pod \"global-pull-secret-syncer-lhtkb\" (UID: \"9255a1d5-2514-441e-b942-6c39843e38f8\") " pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:18:45.710799 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710618 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-socket-dir\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:18:45.710799 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710638 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5mrm\" (UniqueName: \"kubernetes.io/projected/2552d079-24ff-4651-b77e-155af2fe48e9-kube-api-access-t5mrm\") pod \"iptables-alerter-qn24x\" (UID: \"2552d079-24ff-4651-b77e-155af2fe48e9\") " pod="openshift-network-operator/iptables-alerter-qn24x" Oct 31 19:18:45.710799 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710652 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-log-socket\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.710799 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710667 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7whr\" (UniqueName: \"kubernetes.io/projected/81405623-de29-4fc6-bccf-447967669d89-kube-api-access-l7whr\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.710799 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710682 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-run\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.710799 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710698 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-host\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.710799 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710713 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b42xc\" (UniqueName: \"kubernetes.io/projected/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-kube-api-access-b42xc\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.710799 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710754 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/28c13795-a1de-4761-b3fc-feef76ac730d-hosts-file\") pod \"node-resolver-85mq2\" (UID: \"28c13795-a1de-4761-b3fc-feef76ac730d\") " pod="openshift-dns/node-resolver-85mq2" Oct 31 19:18:45.710799 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710775 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mjmg\" (UniqueName: \"kubernetes.io/projected/5708be73-8282-4caa-bce3-46d4b708860e-kube-api-access-9mjmg\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.710799 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710796 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81405623-de29-4fc6-bccf-447967669d89-ovnkube-config\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.711314 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710813 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47cff5a0-505b-45e9-9788-cbe4b557917e-system-cni-dir\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:18:45.711314 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710829 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47cff5a0-505b-45e9-9788-cbe4b557917e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:18:45.711314 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710845 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-run-netns\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.711314 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710883 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-var-lib-openvswitch\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.711314 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710899 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-system-cni-dir\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.711314 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710913 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-var-lib-cni-multus\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.711314 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710933 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-multus-conf-dir\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.711314 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710950 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81405623-de29-4fc6-bccf-447967669d89-ovnkube-script-lib\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.711314 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.710989 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-tmp\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.711314 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711005 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-slash\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.711314 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711021 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-run-netns\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.711314 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711035 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-var-lib-kubelet\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.711314 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711050 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-run-multus-certs\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.711314 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711065 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9255a1d5-2514-441e-b942-6c39843e38f8-dbus\") pod \"global-pull-secret-syncer-lhtkb\" (UID: \"9255a1d5-2514-441e-b942-6c39843e38f8\") " pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:18:45.711314 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711089 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88a1fa4b-1191-434a-8e7c-783163ece24d-metrics-certs\") pod \"network-metrics-daemon-dfbrp\" (UID: \"88a1fa4b-1191-434a-8e7c-783163ece24d\") " pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:18:45.711314 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711109 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-run-systemd\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.711314 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711126 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-run-openvswitch\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.712105 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711140 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81405623-de29-4fc6-bccf-447967669d89-ovn-node-metrics-cert\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.712105 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711155 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47cff5a0-505b-45e9-9788-cbe4b557917e-cni-binary-copy\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:18:45.712105 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711169 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-device-dir\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:18:45.712105 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711191 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-sys-fs\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:18:45.712105 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711212 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnwvq\" (UniqueName: \"kubernetes.io/projected/52432f95-9eb4-4fcd-b795-e475f3399c62-kube-api-access-pnwvq\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:18:45.712105 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711228 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-modprobe-d\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.712105 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711244 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-sysctl-conf\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.712105 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711265 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-tuned\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.712105 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711284 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1d0b685c-252c-459c-9dba-d19c1bc6e5d0-agent-certs\") pod \"konnectivity-agent-wxdtf\" (UID: \"1d0b685c-252c-459c-9dba-d19c1bc6e5d0\") " pod="kube-system/konnectivity-agent-wxdtf" Oct 31 19:18:45.712105 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711300 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jzkh\" (UniqueName: \"kubernetes.io/projected/47cff5a0-505b-45e9-9788-cbe4b557917e-kube-api-access-8jzkh\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:18:45.712105 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711316 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2552d079-24ff-4651-b77e-155af2fe48e9-iptables-alerter-script\") pod \"iptables-alerter-qn24x\" (UID: \"2552d079-24ff-4651-b77e-155af2fe48e9\") " pod="openshift-network-operator/iptables-alerter-qn24x" Oct 31 19:18:45.712105 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711346 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-systemd-units\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.712105 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711363 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5708be73-8282-4caa-bce3-46d4b708860e-cni-binary-copy\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.712105 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711378 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5708be73-8282-4caa-bce3-46d4b708860e-multus-daemon-config\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.712105 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711394 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-etc-kubernetes\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.712105 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711410 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-cni-bin\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.712921 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711425 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-sys\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.712921 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711442 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c13795-a1de-4761-b3fc-feef76ac730d-tmp-dir\") pod \"node-resolver-85mq2\" (UID: \"28c13795-a1de-4761-b3fc-feef76ac730d\") " pod="openshift-dns/node-resolver-85mq2" Oct 31 19:18:45.712921 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711456 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47cff5a0-505b-45e9-9788-cbe4b557917e-cnibin\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:18:45.712921 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711478 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-hostroot\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.712921 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711493 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47cff5a0-505b-45e9-9788-cbe4b557917e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:18:45.712921 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711509 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47cff5a0-505b-45e9-9788-cbe4b557917e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:18:45.712921 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711528 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-registration-dir\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:18:45.712921 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711550 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/84f5548f-7d70-4391-89c9-3bd3c1784717-host\") pod \"node-ca-8m482\" (UID: \"84f5548f-7d70-4391-89c9-3bd3c1784717\") " pod="openshift-image-registry/node-ca-8m482" Oct 31 19:18:45.712921 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711565 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-sysconfig\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.712921 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711582 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.712921 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711597 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-lib-modules\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.712921 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711615 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-var-lib-kubelet\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.712921 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711630 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1d0b685c-252c-459c-9dba-d19c1bc6e5d0-konnectivity-ca\") pod \"konnectivity-agent-wxdtf\" (UID: \"1d0b685c-252c-459c-9dba-d19c1bc6e5d0\") " pod="kube-system/konnectivity-agent-wxdtf" Oct 31 19:18:45.712921 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711647 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-etc-selinux\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:18:45.712921 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711663 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2552d079-24ff-4651-b77e-155af2fe48e9-host-slash\") pod \"iptables-alerter-qn24x\" (UID: \"2552d079-24ff-4651-b77e-155af2fe48e9\") " pod="openshift-network-operator/iptables-alerter-qn24x" Oct 31 19:18:45.712921 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711680 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/84f5548f-7d70-4391-89c9-3bd3c1784717-serviceca\") pod \"node-ca-8m482\" (UID: \"84f5548f-7d70-4391-89c9-3bd3c1784717\") " pod="openshift-image-registry/node-ca-8m482" Oct 31 19:18:45.713727 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711707 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-etc-openvswitch\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.713727 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711724 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd4t7\" (UniqueName: \"kubernetes.io/projected/28c13795-a1de-4761-b3fc-feef76ac730d-kube-api-access-bd4t7\") pod \"node-resolver-85mq2\" (UID: \"28c13795-a1de-4761-b3fc-feef76ac730d\") " pod="openshift-dns/node-resolver-85mq2" Oct 31 19:18:45.713727 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711741 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-kubernetes\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.713727 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711757 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-systemd\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.713727 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711773 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msgz5\" (UniqueName: \"kubernetes.io/projected/84f5548f-7d70-4391-89c9-3bd3c1784717-kube-api-access-msgz5\") pod \"node-ca-8m482\" (UID: \"84f5548f-7d70-4391-89c9-3bd3c1784717\") " pod="openshift-image-registry/node-ca-8m482" Oct 31 19:18:45.713727 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711790 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-run-k8s-cni-cncf-io\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.713727 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711807 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-node-log\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.713727 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711834 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81405623-de29-4fc6-bccf-447967669d89-env-overrides\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.713727 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711853 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:18:45.713727 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711871 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-cnibin\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.713727 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711887 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-kubelet\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.713727 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711906 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-run-ovn-kubernetes\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.713727 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.711923 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q2cm\" (UniqueName: \"kubernetes.io/projected/88a1fa4b-1191-434a-8e7c-783163ece24d-kube-api-access-9q2cm\") pod \"network-metrics-daemon-dfbrp\" (UID: \"88a1fa4b-1191-434a-8e7c-783163ece24d\") " pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:18:45.713727 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.712148 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-multus-socket-dir-parent\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.713727 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.712346 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-multus-cni-dir\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.713727 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.712383 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-var-lib-cni-bin\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.713727 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.712771 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-system-cni-dir\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.714587 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.712816 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-var-lib-cni-multus\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.714587 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.712845 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-multus-conf-dir\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.714587 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.712908 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-run-netns\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.714587 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.712936 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-var-lib-kubelet\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.714587 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.712967 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-run-multus-certs\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.714587 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.713568 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5708be73-8282-4caa-bce3-46d4b708860e-cni-binary-copy\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.714587 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.713668 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-etc-kubernetes\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.714587 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.713749 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-hostroot\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.714587 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.713801 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/84f5548f-7d70-4391-89c9-3bd3c1784717-host\") pod \"node-ca-8m482\" (UID: \"84f5548f-7d70-4391-89c9-3bd3c1784717\") " pod="openshift-image-registry/node-ca-8m482" Oct 31 19:18:45.714587 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.714062 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-run-k8s-cni-cncf-io\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.714587 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.714231 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/84f5548f-7d70-4391-89c9-3bd3c1784717-serviceca\") pod \"node-ca-8m482\" (UID: \"84f5548f-7d70-4391-89c9-3bd3c1784717\") " pod="openshift-image-registry/node-ca-8m482" Oct 31 19:18:45.714587 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.714342 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-cnibin\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.714587 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.714497 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 19:18:45.715317 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.714666 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 19:18:45.715317 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.714699 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 19:18:45.715317 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.714732 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 19:18:45.715317 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.714764 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 19:18:45.715317 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.714790 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 19:18:45.715317 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.714835 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 19:18:45.715317 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.714879 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zwg2q\"" Oct 31 19:18:45.715317 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.714666 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 19:18:45.715317 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.714947 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 19:18:45.715317 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.714987 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 19:18:45.715317 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.715017 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 19:18:45.715317 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.715073 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 19:18:45.716740 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.716716 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cs8fb\"" Oct 31 19:18:45.716838 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.716769 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-j7nlh\"" Oct 31 19:18:45.716916 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.716869 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 19:18:45.716916 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.716909 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 19:18:45.717148 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.717132 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 19:18:45.717236 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.717221 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kbgxr\"" Oct 31 19:18:45.717296 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.717272 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 19:18:45.717349 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.717331 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dcmmt\"" Oct 31 19:18:45.717407 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.717381 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 19:18:45.717466 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.717406 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 19:18:45.719456 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.719434 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bbst6\"" Oct 31 19:18:45.725131 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.725108 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5708be73-8282-4caa-bce3-46d4b708860e-multus-daemon-config\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.727371 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.727345 2566 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 19:18:45.732258 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.732227 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mjmg\" (UniqueName: \"kubernetes.io/projected/5708be73-8282-4caa-bce3-46d4b708860e-kube-api-access-9mjmg\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.736005 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.735944 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msgz5\" (UniqueName: \"kubernetes.io/projected/84f5548f-7d70-4391-89c9-3bd3c1784717-kube-api-access-msgz5\") pod \"node-ca-8m482\" (UID: \"84f5548f-7d70-4391-89c9-3bd3c1784717\") " pod="openshift-image-registry/node-ca-8m482" Oct 31 19:18:45.737346 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.737297 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal" event={"ID":"956815bb47502a7ffa36d27c2bef7876","Type":"ContainerStarted","Data":"b1940201e2acbb0579e67e09eba8ea212ddb141274f55d9b7d9da554b57e149f"} Oct 31 19:18:45.738952 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.738907 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" event={"ID":"2993240a68b51a645ff6b733bc57689d","Type":"ContainerStarted","Data":"a266a13bd4e51c27d828d0bb7e1adbb395a4f2faed8940c3282bc8fa68722f9f"} Oct 31 19:18:45.812548 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.812512 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-run-ovn\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.812739 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.812557 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-cni-netd\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.812739 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.812587 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9255a1d5-2514-441e-b942-6c39843e38f8-original-pull-secret\") pod \"global-pull-secret-syncer-lhtkb\" (UID: \"9255a1d5-2514-441e-b942-6c39843e38f8\") " pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:18:45.812739 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.812613 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7djgh\" (UniqueName: \"kubernetes.io/projected/23a10416-51b1-415f-b2fc-892b4d662b1f-kube-api-access-7djgh\") pod \"network-check-target-5mkxz\" (UID: \"23a10416-51b1-415f-b2fc-892b4d662b1f\") " pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:18:45.812739 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.812643 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-sysctl-d\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.812739 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.812643 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-run-ovn\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.812739 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.812668 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9255a1d5-2514-441e-b942-6c39843e38f8-kubelet-config\") pod \"global-pull-secret-syncer-lhtkb\" (UID: \"9255a1d5-2514-441e-b942-6c39843e38f8\") " pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:18:45.812739 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.812690 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-socket-dir\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:18:45.812739 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.812706 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5mrm\" (UniqueName: \"kubernetes.io/projected/2552d079-24ff-4651-b77e-155af2fe48e9-kube-api-access-t5mrm\") pod \"iptables-alerter-qn24x\" (UID: \"2552d079-24ff-4651-b77e-155af2fe48e9\") " pod="openshift-network-operator/iptables-alerter-qn24x" Oct 31 19:18:45.812739 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.812707 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-cni-netd\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.812739 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.812722 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-log-socket\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.812739 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.812738 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7whr\" (UniqueName: \"kubernetes.io/projected/81405623-de29-4fc6-bccf-447967669d89-kube-api-access-l7whr\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.813273 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.812755 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-run\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.813273 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.812775 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-host\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.813273 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.812862 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-host\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.813273 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:45.812971 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 19:18:45.813273 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813006 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b42xc\" (UniqueName: \"kubernetes.io/projected/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-kube-api-access-b42xc\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.813273 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813052 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-log-socket\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.813273 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:45.813056 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9255a1d5-2514-441e-b942-6c39843e38f8-original-pull-secret podName:9255a1d5-2514-441e-b942-6c39843e38f8 nodeName:}" failed. No retries permitted until 2026-10-31 19:18:46.313012947 +0000 UTC m=+3.127107185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9255a1d5-2514-441e-b942-6c39843e38f8-original-pull-secret") pod "global-pull-secret-syncer-lhtkb" (UID: "9255a1d5-2514-441e-b942-6c39843e38f8") : object "kube-system"/"original-pull-secret" not registered Oct 31 19:18:45.813273 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813091 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-sysctl-d\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.813273 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813100 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/28c13795-a1de-4761-b3fc-feef76ac730d-hosts-file\") pod \"node-resolver-85mq2\" (UID: \"28c13795-a1de-4761-b3fc-feef76ac730d\") " pod="openshift-dns/node-resolver-85mq2" Oct 31 19:18:45.813273 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813123 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-run\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.813273 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813129 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81405623-de29-4fc6-bccf-447967669d89-ovnkube-config\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.813273 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813166 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47cff5a0-505b-45e9-9788-cbe4b557917e-system-cni-dir\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:18:45.813273 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.812753 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9255a1d5-2514-441e-b942-6c39843e38f8-kubelet-config\") pod \"global-pull-secret-syncer-lhtkb\" (UID: \"9255a1d5-2514-441e-b942-6c39843e38f8\") " pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:18:45.813273 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813196 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47cff5a0-505b-45e9-9788-cbe4b557917e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:18:45.813273 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813245 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-run-netns\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.813273 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813257 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/28c13795-a1de-4761-b3fc-feef76ac730d-hosts-file\") pod \"node-resolver-85mq2\" (UID: \"28c13795-a1de-4761-b3fc-feef76ac730d\") " pod="openshift-dns/node-resolver-85mq2" Oct 31 19:18:45.813273 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813272 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-var-lib-openvswitch\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.813936 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813304 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81405623-de29-4fc6-bccf-447967669d89-ovnkube-script-lib\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.813936 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813331 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-tmp\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.813936 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813333 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47cff5a0-505b-45e9-9788-cbe4b557917e-system-cni-dir\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:18:45.813936 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813165 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-socket-dir\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:18:45.813936 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813343 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-var-lib-openvswitch\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.813936 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813305 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-run-netns\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.813936 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813356 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-slash\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.813936 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813406 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47cff5a0-505b-45e9-9788-cbe4b557917e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:18:45.813936 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813419 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-slash\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.813936 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813421 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9255a1d5-2514-441e-b942-6c39843e38f8-dbus\") pod \"global-pull-secret-syncer-lhtkb\" (UID: \"9255a1d5-2514-441e-b942-6c39843e38f8\") " pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:18:45.813936 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813459 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88a1fa4b-1191-434a-8e7c-783163ece24d-metrics-certs\") pod \"network-metrics-daemon-dfbrp\" (UID: \"88a1fa4b-1191-434a-8e7c-783163ece24d\") " pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:18:45.813936 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813490 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-run-systemd\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.813936 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813525 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-run-openvswitch\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.813936 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813559 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81405623-de29-4fc6-bccf-447967669d89-ovn-node-metrics-cert\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.813936 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:45.813590 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:18:45.813936 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813638 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-run-openvswitch\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.813936 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:45.813702 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88a1fa4b-1191-434a-8e7c-783163ece24d-metrics-certs podName:88a1fa4b-1191-434a-8e7c-783163ece24d nodeName:}" failed. No retries permitted until 2026-10-31 19:18:46.313681477 +0000 UTC m=+3.127775694 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88a1fa4b-1191-434a-8e7c-783163ece24d-metrics-certs") pod "network-metrics-daemon-dfbrp" (UID: "88a1fa4b-1191-434a-8e7c-783163ece24d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:18:45.814789 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813700 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-run-systemd\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.814789 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813590 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47cff5a0-505b-45e9-9788-cbe4b557917e-cni-binary-copy\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:18:45.814789 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813767 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-device-dir\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:18:45.814789 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813798 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-sys-fs\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:18:45.814789 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813833 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnwvq\" (UniqueName: \"kubernetes.io/projected/52432f95-9eb4-4fcd-b795-e475f3399c62-kube-api-access-pnwvq\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:18:45.814789 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813867 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-modprobe-d\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.814789 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813897 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-sysctl-conf\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.814789 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813927 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-tuned\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.814789 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.813975 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1d0b685c-252c-459c-9dba-d19c1bc6e5d0-agent-certs\") pod \"konnectivity-agent-wxdtf\" (UID: \"1d0b685c-252c-459c-9dba-d19c1bc6e5d0\") " pod="kube-system/konnectivity-agent-wxdtf" Oct 31 19:18:45.814789 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.814006 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9255a1d5-2514-441e-b942-6c39843e38f8-dbus\") pod \"global-pull-secret-syncer-lhtkb\" (UID: \"9255a1d5-2514-441e-b942-6c39843e38f8\") " pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:18:45.814789 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.814006 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jzkh\" (UniqueName: \"kubernetes.io/projected/47cff5a0-505b-45e9-9788-cbe4b557917e-kube-api-access-8jzkh\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:18:45.814789 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.814049 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2552d079-24ff-4651-b77e-155af2fe48e9-iptables-alerter-script\") pod \"iptables-alerter-qn24x\" (UID: \"2552d079-24ff-4651-b77e-155af2fe48e9\") " pod="openshift-network-operator/iptables-alerter-qn24x" Oct 31 19:18:45.814789 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.814075 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-systemd-units\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.814789 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.814106 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-cni-bin\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.814789 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.814147 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-device-dir\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:18:45.814789 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.814152 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-cni-bin\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.814789 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.814177 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-sys-fs\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:18:45.815578 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.814265 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-sys\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.815578 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.814296 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c13795-a1de-4761-b3fc-feef76ac730d-tmp-dir\") pod \"node-resolver-85mq2\" (UID: \"28c13795-a1de-4761-b3fc-feef76ac730d\") " pod="openshift-dns/node-resolver-85mq2" Oct 31 19:18:45.815578 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.814324 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47cff5a0-505b-45e9-9788-cbe4b557917e-cnibin\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:18:45.815578 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.814333 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47cff5a0-505b-45e9-9788-cbe4b557917e-cni-binary-copy\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:18:45.815578 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.814356 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47cff5a0-505b-45e9-9788-cbe4b557917e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:18:45.815578 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.814366 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81405623-de29-4fc6-bccf-447967669d89-ovnkube-config\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.815578 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.814385 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47cff5a0-505b-45e9-9788-cbe4b557917e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:18:45.815578 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.814433 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81405623-de29-4fc6-bccf-447967669d89-ovnkube-script-lib\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.815578 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.815448 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-sysctl-conf\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.821370 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.815843 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2552d079-24ff-4651-b77e-155af2fe48e9-iptables-alerter-script\") pod \"iptables-alerter-qn24x\" (UID: \"2552d079-24ff-4651-b77e-155af2fe48e9\") " pod="openshift-network-operator/iptables-alerter-qn24x" Oct 31 19:18:45.821370 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.817760 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-tmp\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.821370 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.817935 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-modprobe-d\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.821370 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.818143 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1d0b685c-252c-459c-9dba-d19c1bc6e5d0-agent-certs\") pod \"konnectivity-agent-wxdtf\" (UID: \"1d0b685c-252c-459c-9dba-d19c1bc6e5d0\") " pod="kube-system/konnectivity-agent-wxdtf" Oct 31 19:18:45.821370 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.818243 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47cff5a0-505b-45e9-9788-cbe4b557917e-cnibin\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:18:45.821370 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.818252 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c13795-a1de-4761-b3fc-feef76ac730d-tmp-dir\") pod \"node-resolver-85mq2\" (UID: \"28c13795-a1de-4761-b3fc-feef76ac730d\") " pod="openshift-dns/node-resolver-85mq2" Oct 31 19:18:45.821370 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.818324 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-sys\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.821370 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.818380 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81405623-de29-4fc6-bccf-447967669d89-ovn-node-metrics-cert\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.821370 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.818467 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-systemd-units\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.821370 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.818500 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-tuned\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.821370 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.818509 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-registration-dir\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:18:45.821370 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.818570 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-sysconfig\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.821370 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.818623 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.821370 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.818635 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-registration-dir\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:18:45.821370 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.818660 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-lib-modules\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.821370 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.818696 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-sysconfig\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.821370 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.818704 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47cff5a0-505b-45e9-9788-cbe4b557917e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:18:45.822309 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.818701 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.822309 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.818753 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-var-lib-kubelet\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.822309 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.818792 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1d0b685c-252c-459c-9dba-d19c1bc6e5d0-konnectivity-ca\") pod \"konnectivity-agent-wxdtf\" (UID: \"1d0b685c-252c-459c-9dba-d19c1bc6e5d0\") " pod="kube-system/konnectivity-agent-wxdtf" Oct 31 19:18:45.822309 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.818800 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-var-lib-kubelet\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.822309 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.818821 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-lib-modules\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.822309 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.818829 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-etc-selinux\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:18:45.822309 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.818869 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2552d079-24ff-4651-b77e-155af2fe48e9-host-slash\") pod \"iptables-alerter-qn24x\" (UID: \"2552d079-24ff-4651-b77e-155af2fe48e9\") " pod="openshift-network-operator/iptables-alerter-qn24x" Oct 31 19:18:45.822309 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.818906 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-etc-openvswitch\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.822309 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.818911 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-etc-selinux\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:18:45.822309 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.818950 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2552d079-24ff-4651-b77e-155af2fe48e9-host-slash\") pod \"iptables-alerter-qn24x\" (UID: \"2552d079-24ff-4651-b77e-155af2fe48e9\") " pod="openshift-network-operator/iptables-alerter-qn24x" Oct 31 19:18:45.822309 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.818956 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd4t7\" (UniqueName: \"kubernetes.io/projected/28c13795-a1de-4761-b3fc-feef76ac730d-kube-api-access-bd4t7\") pod \"node-resolver-85mq2\" (UID: \"28c13795-a1de-4761-b3fc-feef76ac730d\") " pod="openshift-dns/node-resolver-85mq2" Oct 31 19:18:45.822309 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.819000 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-kubernetes\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.822309 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.819036 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-systemd\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.822309 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.819073 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-node-log\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.822309 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.819104 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81405623-de29-4fc6-bccf-447967669d89-env-overrides\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.822309 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.819138 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:18:45.822309 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.819175 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-kubelet\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.823199 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.819211 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-run-ovn-kubernetes\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.823199 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.819234 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-etc-openvswitch\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.823199 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.819245 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q2cm\" (UniqueName: \"kubernetes.io/projected/88a1fa4b-1191-434a-8e7c-783163ece24d-kube-api-access-9q2cm\") pod \"network-metrics-daemon-dfbrp\" (UID: \"88a1fa4b-1191-434a-8e7c-783163ece24d\") " pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:18:45.823199 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.819323 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47cff5a0-505b-45e9-9788-cbe4b557917e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:18:45.823199 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.819357 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:18:45.823199 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.819360 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-run-ovn-kubernetes\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.823199 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.819399 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-kubernetes\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.823199 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.819409 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-kubelet\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.823199 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.819414 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-systemd\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.823199 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.819423 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1d0b685c-252c-459c-9dba-d19c1bc6e5d0-konnectivity-ca\") pod \"konnectivity-agent-wxdtf\" (UID: \"1d0b685c-252c-459c-9dba-d19c1bc6e5d0\") " pod="kube-system/konnectivity-agent-wxdtf" Oct 31 19:18:45.823199 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.819456 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-node-log\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.823199 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.819819 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81405623-de29-4fc6-bccf-447967669d89-env-overrides\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.832347 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.832324 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b42xc\" (UniqueName: \"kubernetes.io/projected/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-kube-api-access-b42xc\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.833196 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:45.833173 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 19:18:45.833196 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:45.833199 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 19:18:45.833391 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:45.833214 2566 projected.go:196] Error preparing data for projected volume kube-api-access-7djgh for pod openshift-network-diagnostics/network-check-target-5mkxz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:18:45.833391 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:45.833280 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23a10416-51b1-415f-b2fc-892b4d662b1f-kube-api-access-7djgh podName:23a10416-51b1-415f-b2fc-892b4d662b1f nodeName:}" failed. No retries permitted until 2026-10-31 19:18:46.333262122 +0000 UTC m=+3.147356350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7djgh" (UniqueName: "kubernetes.io/projected/23a10416-51b1-415f-b2fc-892b4d662b1f-kube-api-access-7djgh") pod "network-check-target-5mkxz" (UID: "23a10416-51b1-415f-b2fc-892b4d662b1f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:18:45.835392 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.835369 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7whr\" (UniqueName: \"kubernetes.io/projected/81405623-de29-4fc6-bccf-447967669d89-kube-api-access-l7whr\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.841284 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.841259 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5mrm\" (UniqueName: \"kubernetes.io/projected/2552d079-24ff-4651-b77e-155af2fe48e9-kube-api-access-t5mrm\") pod \"iptables-alerter-qn24x\" (UID: \"2552d079-24ff-4651-b77e-155af2fe48e9\") " pod="openshift-network-operator/iptables-alerter-qn24x" Oct 31 19:18:45.847131 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.847111 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnwvq\" (UniqueName: \"kubernetes.io/projected/52432f95-9eb4-4fcd-b795-e475f3399c62-kube-api-access-pnwvq\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:18:45.849790 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.849764 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q2cm\" (UniqueName: \"kubernetes.io/projected/88a1fa4b-1191-434a-8e7c-783163ece24d-kube-api-access-9q2cm\") pod \"network-metrics-daemon-dfbrp\" (UID: \"88a1fa4b-1191-434a-8e7c-783163ece24d\") " pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:18:45.850494 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.850475 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd4t7\" (UniqueName: \"kubernetes.io/projected/28c13795-a1de-4761-b3fc-feef76ac730d-kube-api-access-bd4t7\") pod \"node-resolver-85mq2\" (UID: \"28c13795-a1de-4761-b3fc-feef76ac730d\") " pod="openshift-dns/node-resolver-85mq2" Oct 31 19:18:45.850591 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.850508 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jzkh\" (UniqueName: \"kubernetes.io/projected/47cff5a0-505b-45e9-9788-cbe4b557917e-kube-api-access-8jzkh\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:18:45.913228 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.913194 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8m482" Oct 31 19:18:45.922862 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.922664 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tm5fh" Oct 31 19:18:45.926148 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:45.926118 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f5548f_7d70_4391_89c9_3bd3c1784717.slice/crio-208233d55fdbbef2a6b3722af35e78145ae9fa2022b91872c59d62678b9e4439 WatchSource:0}: Error finding container 208233d55fdbbef2a6b3722af35e78145ae9fa2022b91872c59d62678b9e4439: Status 404 returned error can't find the container with id 208233d55fdbbef2a6b3722af35e78145ae9fa2022b91872c59d62678b9e4439 Oct 31 19:18:45.933670 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:45.933646 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5708be73_8282_4caa_bce3_46d4b708860e.slice/crio-cdc79ea0e4f25f828ac0bf53e857206d62cee2ede1b798d52114280ccd5151fe WatchSource:0}: Error finding container cdc79ea0e4f25f828ac0bf53e857206d62cee2ede1b798d52114280ccd5151fe: Status 404 returned error can't find the container with id cdc79ea0e4f25f828ac0bf53e857206d62cee2ede1b798d52114280ccd5151fe Oct 31 19:18:45.953693 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.953663 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:18:45.958217 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.958193 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-85mq2" Oct 31 19:18:45.961601 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.961578 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:18:45.964625 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:45.964601 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf30e3e5_bfd9_4f93_9fe9_1514562c4267.slice/crio-330a197c14ab15f2f6a438cb417717b44b4346dbfda8a411bd04fd398249231a WatchSource:0}: Error finding container 330a197c14ab15f2f6a438cb417717b44b4346dbfda8a411bd04fd398249231a: Status 404 returned error can't find the container with id 330a197c14ab15f2f6a438cb417717b44b4346dbfda8a411bd04fd398249231a Oct 31 19:18:45.973723 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:45.973694 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c13795_a1de_4761_b3fc_feef76ac730d.slice/crio-ed192e0a5ee892c52cf2f4eefab17503ef6d0778bf98383753a44830616469b2 WatchSource:0}: Error finding container ed192e0a5ee892c52cf2f4eefab17503ef6d0778bf98383753a44830616469b2: Status 404 returned error can't find the container with id ed192e0a5ee892c52cf2f4eefab17503ef6d0778bf98383753a44830616469b2 Oct 31 19:18:45.981662 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.981514 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wxdtf" Oct 31 19:18:45.983477 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.983458 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:18:45.983834 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:45.983812 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81405623_de29_4fc6_bccf_447967669d89.slice/crio-51292dc6a2394b3eccfbe90c1da601bf466be77c7db60e713d44b8d61403a6ff WatchSource:0}: Error finding container 51292dc6a2394b3eccfbe90c1da601bf466be77c7db60e713d44b8d61403a6ff: Status 404 returned error can't find the container with id 51292dc6a2394b3eccfbe90c1da601bf466be77c7db60e713d44b8d61403a6ff Oct 31 19:18:45.985807 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.985786 2566 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:18:45.992464 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.992444 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:18:45.994186 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:45.994165 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qn24x" Oct 31 19:18:46.001368 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:46.001327 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0b685c_252c_459c_9dba_d19c1bc6e5d0.slice/crio-b9874207c5992fedc22950b400fa8eab1e93572799123f4c1b8a9a0085a9f2a1 WatchSource:0}: Error finding container b9874207c5992fedc22950b400fa8eab1e93572799123f4c1b8a9a0085a9f2a1: Status 404 returned error can't find the container with id b9874207c5992fedc22950b400fa8eab1e93572799123f4c1b8a9a0085a9f2a1 Oct 31 19:18:46.007829 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:18:46.007801 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47cff5a0_505b_45e9_9788_cbe4b557917e.slice/crio-cb43e211e12624c829c6b1af0b205736d0fd4f1722fccd5a813ad06fe34e47ba WatchSource:0}: Error finding container cb43e211e12624c829c6b1af0b205736d0fd4f1722fccd5a813ad06fe34e47ba: Status 404 returned error can't find the container with id cb43e211e12624c829c6b1af0b205736d0fd4f1722fccd5a813ad06fe34e47ba Oct 31 19:18:46.322165 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:46.322135 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9255a1d5-2514-441e-b942-6c39843e38f8-original-pull-secret\") pod \"global-pull-secret-syncer-lhtkb\" (UID: \"9255a1d5-2514-441e-b942-6c39843e38f8\") " pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:18:46.322335 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:46.322205 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88a1fa4b-1191-434a-8e7c-783163ece24d-metrics-certs\") pod \"network-metrics-daemon-dfbrp\" (UID: \"88a1fa4b-1191-434a-8e7c-783163ece24d\") " pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:18:46.322335 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:46.322316 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:18:46.322452 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:46.322376 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 19:18:46.322452 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:46.322405 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88a1fa4b-1191-434a-8e7c-783163ece24d-metrics-certs podName:88a1fa4b-1191-434a-8e7c-783163ece24d nodeName:}" failed. No retries permitted until 2026-10-31 19:18:47.322383372 +0000 UTC m=+4.136477605 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88a1fa4b-1191-434a-8e7c-783163ece24d-metrics-certs") pod "network-metrics-daemon-dfbrp" (UID: "88a1fa4b-1191-434a-8e7c-783163ece24d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:18:46.322452 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:46.322437 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9255a1d5-2514-441e-b942-6c39843e38f8-original-pull-secret podName:9255a1d5-2514-441e-b942-6c39843e38f8 nodeName:}" failed. No retries permitted until 2026-10-31 19:18:47.322419442 +0000 UTC m=+4.136513675 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9255a1d5-2514-441e-b942-6c39843e38f8-original-pull-secret") pod "global-pull-secret-syncer-lhtkb" (UID: "9255a1d5-2514-441e-b942-6c39843e38f8") : object "kube-system"/"original-pull-secret" not registered Oct 31 19:18:46.422832 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:46.422800 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7djgh\" (UniqueName: \"kubernetes.io/projected/23a10416-51b1-415f-b2fc-892b4d662b1f-kube-api-access-7djgh\") pod \"network-check-target-5mkxz\" (UID: \"23a10416-51b1-415f-b2fc-892b4d662b1f\") " pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:18:46.423050 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:46.423027 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 19:18:46.423121 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:46.423057 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 19:18:46.423121 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:46.423070 2566 projected.go:196] Error preparing data for projected volume kube-api-access-7djgh for pod openshift-network-diagnostics/network-check-target-5mkxz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:18:46.423231 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:46.423144 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23a10416-51b1-415f-b2fc-892b4d662b1f-kube-api-access-7djgh podName:23a10416-51b1-415f-b2fc-892b4d662b1f nodeName:}" failed. No retries permitted until 2026-10-31 19:18:47.423124227 +0000 UTC m=+4.237218453 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7djgh" (UniqueName: "kubernetes.io/projected/23a10416-51b1-415f-b2fc-892b4d662b1f-kube-api-access-7djgh") pod "network-check-target-5mkxz" (UID: "23a10416-51b1-415f-b2fc-892b4d662b1f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:18:46.666721 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:46.666651 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 19:13:45 +0000 UTC" deadline="2028-07-19 07:47:23.422404292 +0000 UTC" Oct 31 19:18:46.666721 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:46.666722 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15036h28m36.755687118s" Oct 31 19:18:46.731681 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:46.731647 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:18:46.731858 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:46.731833 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtkb" podUID="9255a1d5-2514-441e-b942-6c39843e38f8" Oct 31 19:18:46.744173 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:46.744133 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qn24x" event={"ID":"2552d079-24ff-4651-b77e-155af2fe48e9","Type":"ContainerStarted","Data":"c94c48e8e9113d38dfd2e937b8021509e9458f0e33d347e220d5e27a246dbf95"} Oct 31 19:18:46.747440 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:46.747354 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bhqht" event={"ID":"47cff5a0-505b-45e9-9788-cbe4b557917e","Type":"ContainerStarted","Data":"cb43e211e12624c829c6b1af0b205736d0fd4f1722fccd5a813ad06fe34e47ba"} Oct 31 19:18:46.772112 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:46.772077 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wxdtf" event={"ID":"1d0b685c-252c-459c-9dba-d19c1bc6e5d0","Type":"ContainerStarted","Data":"b9874207c5992fedc22950b400fa8eab1e93572799123f4c1b8a9a0085a9f2a1"} Oct 31 19:18:46.788299 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:46.782475 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" event={"ID":"81405623-de29-4fc6-bccf-447967669d89","Type":"ContainerStarted","Data":"51292dc6a2394b3eccfbe90c1da601bf466be77c7db60e713d44b8d61403a6ff"} Oct 31 19:18:46.792087 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:46.792051 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-85mq2" event={"ID":"28c13795-a1de-4761-b3fc-feef76ac730d","Type":"ContainerStarted","Data":"ed192e0a5ee892c52cf2f4eefab17503ef6d0778bf98383753a44830616469b2"} Oct 31 19:18:46.797964 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:46.794845 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" event={"ID":"bf30e3e5-bfd9-4f93-9fe9-1514562c4267","Type":"ContainerStarted","Data":"330a197c14ab15f2f6a438cb417717b44b4346dbfda8a411bd04fd398249231a"} Oct 31 19:18:46.801554 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:46.801518 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" event={"ID":"52432f95-9eb4-4fcd-b795-e475f3399c62","Type":"ContainerStarted","Data":"24cf43bbeef6d1006e432a1b0d9d36c4bcb020d1ecca07a90c945a69686c1097"} Oct 31 19:18:46.807243 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:46.807219 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tm5fh" event={"ID":"5708be73-8282-4caa-bce3-46d4b708860e","Type":"ContainerStarted","Data":"cdc79ea0e4f25f828ac0bf53e857206d62cee2ede1b798d52114280ccd5151fe"} Oct 31 19:18:46.808820 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:46.808766 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8m482" event={"ID":"84f5548f-7d70-4391-89c9-3bd3c1784717","Type":"ContainerStarted","Data":"208233d55fdbbef2a6b3722af35e78145ae9fa2022b91872c59d62678b9e4439"} Oct 31 19:18:47.308389 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:47.308357 2566 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:18:47.329314 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:47.329270 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9255a1d5-2514-441e-b942-6c39843e38f8-original-pull-secret\") pod \"global-pull-secret-syncer-lhtkb\" (UID: \"9255a1d5-2514-441e-b942-6c39843e38f8\") " pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:18:47.329480 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:47.329344 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88a1fa4b-1191-434a-8e7c-783163ece24d-metrics-certs\") pod \"network-metrics-daemon-dfbrp\" (UID: \"88a1fa4b-1191-434a-8e7c-783163ece24d\") " pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:18:47.329480 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:47.329385 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 19:18:47.329480 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:47.329449 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9255a1d5-2514-441e-b942-6c39843e38f8-original-pull-secret podName:9255a1d5-2514-441e-b942-6c39843e38f8 nodeName:}" failed. No retries permitted until 2026-10-31 19:18:49.329429936 +0000 UTC m=+6.143524162 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9255a1d5-2514-441e-b942-6c39843e38f8-original-pull-secret") pod "global-pull-secret-syncer-lhtkb" (UID: "9255a1d5-2514-441e-b942-6c39843e38f8") : object "kube-system"/"original-pull-secret" not registered Oct 31 19:18:47.329480 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:47.329457 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:18:47.329697 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:47.329533 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88a1fa4b-1191-434a-8e7c-783163ece24d-metrics-certs podName:88a1fa4b-1191-434a-8e7c-783163ece24d nodeName:}" failed. No retries permitted until 2026-10-31 19:18:49.329517739 +0000 UTC m=+6.143611963 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88a1fa4b-1191-434a-8e7c-783163ece24d-metrics-certs") pod "network-metrics-daemon-dfbrp" (UID: "88a1fa4b-1191-434a-8e7c-783163ece24d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:18:47.430190 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:47.430159 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7djgh\" (UniqueName: \"kubernetes.io/projected/23a10416-51b1-415f-b2fc-892b4d662b1f-kube-api-access-7djgh\") pod \"network-check-target-5mkxz\" (UID: \"23a10416-51b1-415f-b2fc-892b4d662b1f\") " pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:18:47.430314 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:47.430305 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 19:18:47.430366 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:47.430328 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 19:18:47.430366 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:47.430341 2566 projected.go:196] Error preparing data for projected volume kube-api-access-7djgh for pod openshift-network-diagnostics/network-check-target-5mkxz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:18:47.430477 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:47.430390 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23a10416-51b1-415f-b2fc-892b4d662b1f-kube-api-access-7djgh podName:23a10416-51b1-415f-b2fc-892b4d662b1f nodeName:}" failed. No retries permitted until 2026-10-31 19:18:49.43037455 +0000 UTC m=+6.244468765 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7djgh" (UniqueName: "kubernetes.io/projected/23a10416-51b1-415f-b2fc-892b4d662b1f-kube-api-access-7djgh") pod "network-check-target-5mkxz" (UID: "23a10416-51b1-415f-b2fc-892b4d662b1f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:18:47.667298 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:47.667046 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 19:13:45 +0000 UTC" deadline="2028-04-19 23:33:12.716484963 +0000 UTC" Oct 31 19:18:47.667298 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:47.667298 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12868h14m25.049191736s" Oct 31 19:18:47.743391 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:47.736224 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:18:47.743391 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:47.736351 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5mkxz" podUID="23a10416-51b1-415f-b2fc-892b4d662b1f" Oct 31 19:18:47.743391 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:47.736746 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:18:47.743391 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:47.736923 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfbrp" podUID="88a1fa4b-1191-434a-8e7c-783163ece24d" Oct 31 19:18:47.824858 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:47.824075 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal" event={"ID":"956815bb47502a7ffa36d27c2bef7876","Type":"ContainerStarted","Data":"949ec3e3746b843fe26e624b691d3e5fca870d1d89d8c23e6ead0b6a1317bf41"} Oct 31 19:18:47.832961 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:47.832772 2566 generic.go:356] "Generic (PLEG): container finished" podID="2993240a68b51a645ff6b733bc57689d" containerID="322797fac2c246960ad190be1a22df3a0351675e20897170d6b3448082e77c96" exitCode=0 Oct 31 19:18:47.832961 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:47.832812 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" event={"ID":"2993240a68b51a645ff6b733bc57689d","Type":"ContainerDied","Data":"322797fac2c246960ad190be1a22df3a0351675e20897170d6b3448082e77c96"} Oct 31 19:18:47.873652 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:47.873440 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal" podStartSLOduration=2.873426653 podStartE2EDuration="2.873426653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:18:47.845888231 +0000 UTC m=+4.659982463" watchObservedRunningTime="2026-10-31 19:18:47.873426653 +0000 UTC m=+4.687520925" Oct 31 19:18:48.731122 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:48.731090 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:18:48.731557 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:48.731230 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtkb" podUID="9255a1d5-2514-441e-b942-6c39843e38f8" Oct 31 19:18:48.864335 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:48.862996 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" event={"ID":"2993240a68b51a645ff6b733bc57689d","Type":"ContainerStarted","Data":"9ae0dc728cfb6d94d801453920d29f33e7bfbf6f49895960c605cf0774740d74"} Oct 31 19:18:49.348202 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:49.347448 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9255a1d5-2514-441e-b942-6c39843e38f8-original-pull-secret\") pod \"global-pull-secret-syncer-lhtkb\" (UID: \"9255a1d5-2514-441e-b942-6c39843e38f8\") " pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:18:49.348202 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:49.347526 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88a1fa4b-1191-434a-8e7c-783163ece24d-metrics-certs\") pod \"network-metrics-daemon-dfbrp\" (UID: \"88a1fa4b-1191-434a-8e7c-783163ece24d\") " pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:18:49.348202 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:49.347651 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:18:49.348202 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:49.347710 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88a1fa4b-1191-434a-8e7c-783163ece24d-metrics-certs podName:88a1fa4b-1191-434a-8e7c-783163ece24d nodeName:}" failed. No retries permitted until 2026-10-31 19:18:53.347689943 +0000 UTC m=+10.161784173 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88a1fa4b-1191-434a-8e7c-783163ece24d-metrics-certs") pod "network-metrics-daemon-dfbrp" (UID: "88a1fa4b-1191-434a-8e7c-783163ece24d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:18:49.348202 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:49.348112 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 19:18:49.348202 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:49.348161 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9255a1d5-2514-441e-b942-6c39843e38f8-original-pull-secret podName:9255a1d5-2514-441e-b942-6c39843e38f8 nodeName:}" failed. No retries permitted until 2026-10-31 19:18:53.348147275 +0000 UTC m=+10.162241500 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9255a1d5-2514-441e-b942-6c39843e38f8-original-pull-secret") pod "global-pull-secret-syncer-lhtkb" (UID: "9255a1d5-2514-441e-b942-6c39843e38f8") : object "kube-system"/"original-pull-secret" not registered Oct 31 19:18:49.448770 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:49.448234 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7djgh\" (UniqueName: \"kubernetes.io/projected/23a10416-51b1-415f-b2fc-892b4d662b1f-kube-api-access-7djgh\") pod \"network-check-target-5mkxz\" (UID: \"23a10416-51b1-415f-b2fc-892b4d662b1f\") " pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:18:49.448770 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:49.448374 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 19:18:49.448770 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:49.448392 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 19:18:49.448770 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:49.448404 2566 projected.go:196] Error preparing data for projected volume kube-api-access-7djgh for pod openshift-network-diagnostics/network-check-target-5mkxz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:18:49.448770 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:49.448444 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23a10416-51b1-415f-b2fc-892b4d662b1f-kube-api-access-7djgh podName:23a10416-51b1-415f-b2fc-892b4d662b1f nodeName:}" failed. No retries permitted until 2026-10-31 19:18:53.44843216 +0000 UTC m=+10.262526369 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7djgh" (UniqueName: "kubernetes.io/projected/23a10416-51b1-415f-b2fc-892b4d662b1f-kube-api-access-7djgh") pod "network-check-target-5mkxz" (UID: "23a10416-51b1-415f-b2fc-892b4d662b1f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:18:49.734125 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:49.733822 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:18:49.735300 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:49.734775 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfbrp" podUID="88a1fa4b-1191-434a-8e7c-783163ece24d" Oct 31 19:18:49.735300 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:49.735147 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:18:49.735300 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:49.735224 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5mkxz" podUID="23a10416-51b1-415f-b2fc-892b4d662b1f" Oct 31 19:18:49.868320 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:49.868288 2566 generic.go:356] "Generic (PLEG): container finished" podID="47cff5a0-505b-45e9-9788-cbe4b557917e" containerID="7cb2fefa3e14f2c331683970309db8de56404d20b534f9e32e7029a6569daec1" exitCode=0 Oct 31 19:18:49.869356 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:49.869330 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bhqht" event={"ID":"47cff5a0-505b-45e9-9788-cbe4b557917e","Type":"ContainerDied","Data":"7cb2fefa3e14f2c331683970309db8de56404d20b534f9e32e7029a6569daec1"} Oct 31 19:18:49.903082 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:49.902520 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" podStartSLOduration=4.902498464 podStartE2EDuration="4.902498464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:18:48.882861026 +0000 UTC m=+5.696955237" watchObservedRunningTime="2026-10-31 19:18:49.902498464 +0000 UTC m=+6.716592698" Oct 31 19:18:50.732265 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:50.731714 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:18:50.732265 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:50.731890 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtkb" podUID="9255a1d5-2514-441e-b942-6c39843e38f8" Oct 31 19:18:50.874287 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:50.874244 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wxdtf" event={"ID":"1d0b685c-252c-459c-9dba-d19c1bc6e5d0","Type":"ContainerStarted","Data":"ac3046b9909acd1e17006bc82ce5883d933bd16f88e21162721a8c84df83db4f"} Oct 31 19:18:51.733380 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:51.732868 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:18:51.733380 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:51.733009 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:18:51.733380 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:51.733004 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfbrp" podUID="88a1fa4b-1191-434a-8e7c-783163ece24d" Oct 31 19:18:51.733380 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:51.733085 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5mkxz" podUID="23a10416-51b1-415f-b2fc-892b4d662b1f" Oct 31 19:18:51.880880 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:51.880845 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8m482" event={"ID":"84f5548f-7d70-4391-89c9-3bd3c1784717","Type":"ContainerStarted","Data":"6648cfd03014f0376ad1cb396d13bbc896f1341812e2a8150a6fa2cf5a976124"} Oct 31 19:18:51.898836 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:51.898782 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wxdtf" podStartSLOduration=4.033511181 podStartE2EDuration="7.898764365s" totalImagesPullingTime="3.865253184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:18:50.897994461 +0000 UTC m=+7.712088695" watchObservedRunningTime="2026-10-31 19:18:51.898764365 +0000 UTC m=+8.712858596" Oct 31 19:18:51.899030 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:51.898997 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8m482" podStartSLOduration=3.726614874 podStartE2EDuration="8.898988884s" totalImagesPullingTime="5.17237401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:18:51.898551079 +0000 UTC m=+8.712645311" watchObservedRunningTime="2026-10-31 19:18:51.898988884 +0000 UTC m=+8.713083115" Oct 31 19:18:52.464119 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:52.463864 2566 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wxdtf" Oct 31 19:18:52.464673 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:52.464495 2566 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wxdtf" Oct 31 19:18:52.731489 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:52.730949 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:18:52.731489 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:52.731075 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtkb" podUID="9255a1d5-2514-441e-b942-6c39843e38f8" Oct 31 19:18:52.885856 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:52.885819 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" event={"ID":"52432f95-9eb4-4fcd-b795-e475f3399c62","Type":"ContainerStarted","Data":"f80237258db51803e3ba12a99f9abaaff219d9b16b82a9445ed677b37431f202"} Oct 31 19:18:53.381349 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:53.380511 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9255a1d5-2514-441e-b942-6c39843e38f8-original-pull-secret\") pod \"global-pull-secret-syncer-lhtkb\" (UID: \"9255a1d5-2514-441e-b942-6c39843e38f8\") " pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:18:53.381349 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:53.380596 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88a1fa4b-1191-434a-8e7c-783163ece24d-metrics-certs\") pod \"network-metrics-daemon-dfbrp\" (UID: \"88a1fa4b-1191-434a-8e7c-783163ece24d\") " pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:18:53.381349 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:53.380755 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:18:53.381349 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:53.380824 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88a1fa4b-1191-434a-8e7c-783163ece24d-metrics-certs podName:88a1fa4b-1191-434a-8e7c-783163ece24d nodeName:}" failed. No retries permitted until 2026-10-31 19:19:01.380802667 +0000 UTC m=+18.194896902 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88a1fa4b-1191-434a-8e7c-783163ece24d-metrics-certs") pod "network-metrics-daemon-dfbrp" (UID: "88a1fa4b-1191-434a-8e7c-783163ece24d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:18:53.381349 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:53.381253 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 19:18:53.381349 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:53.381310 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9255a1d5-2514-441e-b942-6c39843e38f8-original-pull-secret podName:9255a1d5-2514-441e-b942-6c39843e38f8 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:01.381294287 +0000 UTC m=+18.195388509 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9255a1d5-2514-441e-b942-6c39843e38f8-original-pull-secret") pod "global-pull-secret-syncer-lhtkb" (UID: "9255a1d5-2514-441e-b942-6c39843e38f8") : object "kube-system"/"original-pull-secret" not registered Oct 31 19:18:53.482557 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:53.481924 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7djgh\" (UniqueName: \"kubernetes.io/projected/23a10416-51b1-415f-b2fc-892b4d662b1f-kube-api-access-7djgh\") pod \"network-check-target-5mkxz\" (UID: \"23a10416-51b1-415f-b2fc-892b4d662b1f\") " pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:18:53.482557 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:53.482099 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 19:18:53.482557 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:53.482122 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 19:18:53.482557 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:53.482134 2566 projected.go:196] Error preparing data for projected volume kube-api-access-7djgh for pod openshift-network-diagnostics/network-check-target-5mkxz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:18:53.482557 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:53.482195 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23a10416-51b1-415f-b2fc-892b4d662b1f-kube-api-access-7djgh podName:23a10416-51b1-415f-b2fc-892b4d662b1f nodeName:}" failed. No retries permitted until 2026-10-31 19:19:01.482179267 +0000 UTC m=+18.296273483 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7djgh" (UniqueName: "kubernetes.io/projected/23a10416-51b1-415f-b2fc-892b4d662b1f-kube-api-access-7djgh") pod "network-check-target-5mkxz" (UID: "23a10416-51b1-415f-b2fc-892b4d662b1f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:18:53.732189 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:53.732160 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:18:53.732320 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:53.732297 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfbrp" podUID="88a1fa4b-1191-434a-8e7c-783163ece24d" Oct 31 19:18:53.732462 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:53.732413 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:18:53.732519 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:53.732505 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5mkxz" podUID="23a10416-51b1-415f-b2fc-892b4d662b1f" Oct 31 19:18:53.864707 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:53.864678 2566 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 19:18:53.888883 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:53.888849 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" event={"ID":"52432f95-9eb4-4fcd-b795-e475f3399c62","Type":"ContainerStarted","Data":"0a8784e0060f34d8b715181396d77f3ac7e9b3b8daba46eb78662c24c28e2af4"} Oct 31 19:18:53.890492 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:53.890410 2566 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 19:18:53.891314 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:53.891277 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tm5fh" event={"ID":"5708be73-8282-4caa-bce3-46d4b708860e","Type":"ContainerStarted","Data":"28bdffb28f5d053e31f29ac9edfa742889786fb2f2ec93c64ce561394cbef3c1"} Oct 31 19:18:54.667695 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:54.667347 2566 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T19:18:53.864700708Z","UUID":"074415b3-f50a-4102-bb9c-bbd9512e9559","Handler":null,"Name":"","Endpoint":""} Oct 31 19:18:54.669120 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:54.669091 2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 19:18:54.669120 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:54.669121 2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 19:18:54.731454 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:54.731422 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:18:54.731621 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:54.731559 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtkb" podUID="9255a1d5-2514-441e-b942-6c39843e38f8" Oct 31 19:18:54.895225 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:54.895187 2566 generic.go:356] "Generic (PLEG): container finished" podID="47cff5a0-505b-45e9-9788-cbe4b557917e" containerID="ec22f0135a308db2eb1de7e2573b618d4858fb4b9b9647e9cd0981ec1e94b62a" exitCode=0 Oct 31 19:18:54.895684 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:54.895289 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bhqht" event={"ID":"47cff5a0-505b-45e9-9788-cbe4b557917e","Type":"ContainerDied","Data":"ec22f0135a308db2eb1de7e2573b618d4858fb4b9b9647e9cd0981ec1e94b62a"} Oct 31 19:18:54.897275 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:54.897247 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-85mq2" event={"ID":"28c13795-a1de-4761-b3fc-feef76ac730d","Type":"ContainerStarted","Data":"057f538a18b942837733fd567262f7f2ba325d22562ad6c62d2bdb289574397f"} Oct 31 19:18:54.969702 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:54.969602 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-tm5fh" podStartSLOduration=4.165138435 podStartE2EDuration="11.969582901s" totalImagesPullingTime="7.804444466s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:18:53.920750308 +0000 UTC m=+10.734844542" watchObservedRunningTime="2026-10-31 19:18:54.969582901 +0000 UTC m=+11.783677133" Oct 31 19:18:55.733241 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:55.733213 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:18:55.733400 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:55.733213 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:18:55.733400 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:55.733334 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfbrp" podUID="88a1fa4b-1191-434a-8e7c-783163ece24d" Oct 31 19:18:55.733512 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:55.733408 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5mkxz" podUID="23a10416-51b1-415f-b2fc-892b4d662b1f" Oct 31 19:18:55.902241 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:55.902165 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qn24x" event={"ID":"2552d079-24ff-4651-b77e-155af2fe48e9","Type":"ContainerStarted","Data":"3a05bee243f6a5795409356cb840bb91d3ab2f8d7d80635a537ae01284926982"} Oct 31 19:18:55.905765 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:55.905733 2566 generic.go:356] "Generic (PLEG): container finished" podID="47cff5a0-505b-45e9-9788-cbe4b557917e" containerID="d69a8fa8428e8bc841753a2ce0f3990f2929a6f508f726cb41967e9ed29fb59b" exitCode=0 Oct 31 19:18:55.905906 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:55.905837 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bhqht" event={"ID":"47cff5a0-505b-45e9-9788-cbe4b557917e","Type":"ContainerDied","Data":"d69a8fa8428e8bc841753a2ce0f3990f2929a6f508f726cb41967e9ed29fb59b"} Oct 31 19:18:55.909370 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:55.909312 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" event={"ID":"52432f95-9eb4-4fcd-b795-e475f3399c62","Type":"ContainerStarted","Data":"3ac175fb6e2214cd56d5bab863064190df678c7ad4e190c26a8b468feaa6b16a"} Oct 31 19:18:55.921321 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:55.921268 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qn24x" podStartSLOduration=3.77367971 podStartE2EDuration="11.921250757s" totalImagesPullingTime="8.147571047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:18:55.920775688 +0000 UTC m=+12.734869919" watchObservedRunningTime="2026-10-31 19:18:55.921250757 +0000 UTC m=+12.735344990" Oct 31 19:18:55.921691 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:55.921661 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-85mq2" podStartSLOduration=4.168080875 podStartE2EDuration="11.921648646s" totalImagesPullingTime="7.753567771s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:18:55.002168795 +0000 UTC m=+11.816263025" watchObservedRunningTime="2026-10-31 19:18:55.921648646 +0000 UTC m=+12.735742878" Oct 31 19:18:55.968302 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:55.968243 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" podStartSLOduration=3.3771383569999998 podStartE2EDuration="11.968223892s" totalImagesPullingTime="8.591085535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:18:55.967831406 +0000 UTC m=+12.781925637" watchObservedRunningTime="2026-10-31 19:18:55.968223892 +0000 UTC m=+12.782318124" Oct 31 19:18:56.730845 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:56.730777 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:18:56.731001 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:56.730887 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtkb" podUID="9255a1d5-2514-441e-b942-6c39843e38f8" Oct 31 19:18:56.913343 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:56.913310 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" event={"ID":"bf30e3e5-bfd9-4f93-9fe9-1514562c4267","Type":"ContainerStarted","Data":"3903c2e4a1b910932376e47fe62a0786cbaca96e95aabae81ed72821be4c5c97"} Oct 31 19:18:56.934943 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:56.934892 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" podStartSLOduration=2.5066391550000002 podStartE2EDuration="12.934879358s" totalImagesPullingTime="10.428240203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:18:56.934794875 +0000 UTC m=+13.748889106" watchObservedRunningTime="2026-10-31 19:18:56.934879358 +0000 UTC m=+13.748973588" Oct 31 19:18:57.730789 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:57.730761 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:18:57.730980 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:57.730896 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfbrp" podUID="88a1fa4b-1191-434a-8e7c-783163ece24d" Oct 31 19:18:57.730980 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:57.730932 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:18:57.731087 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:57.731031 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5mkxz" podUID="23a10416-51b1-415f-b2fc-892b4d662b1f" Oct 31 19:18:57.918262 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:57.918224 2566 generic.go:356] "Generic (PLEG): container finished" podID="47cff5a0-505b-45e9-9788-cbe4b557917e" containerID="bfaa565d5f19ee223e2a41a44bcce5d1fc2baf3a38e87e9b32a1954d19d107b5" exitCode=0 Oct 31 19:18:57.918892 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:57.918319 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bhqht" event={"ID":"47cff5a0-505b-45e9-9788-cbe4b557917e","Type":"ContainerDied","Data":"bfaa565d5f19ee223e2a41a44bcce5d1fc2baf3a38e87e9b32a1954d19d107b5"} Oct 31 19:18:58.731523 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:58.731315 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:18:58.731682 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:58.731609 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtkb" podUID="9255a1d5-2514-441e-b942-6c39843e38f8" Oct 31 19:18:59.731237 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:59.731203 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:18:59.731237 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:59.731239 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:18:59.731767 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:59.731335 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfbrp" podUID="88a1fa4b-1191-434a-8e7c-783163ece24d" Oct 31 19:18:59.731767 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:18:59.731414 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5mkxz" podUID="23a10416-51b1-415f-b2fc-892b4d662b1f" Oct 31 19:18:59.926091 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:18:59.926063 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bhqht" event={"ID":"47cff5a0-505b-45e9-9788-cbe4b557917e","Type":"ContainerStarted","Data":"94d29bf03388cbe8d79beb93341418e8ccfd1d6bfeb09089dec603d19dbff25b"} Oct 31 19:19:00.731426 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:00.731389 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:19:00.731837 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:00.731499 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtkb" podUID="9255a1d5-2514-441e-b942-6c39843e38f8" Oct 31 19:19:00.930755 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:00.930730 2566 generic.go:356] "Generic (PLEG): container finished" podID="47cff5a0-505b-45e9-9788-cbe4b557917e" containerID="94d29bf03388cbe8d79beb93341418e8ccfd1d6bfeb09089dec603d19dbff25b" exitCode=0 Oct 31 19:19:00.930857 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:00.930818 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bhqht" event={"ID":"47cff5a0-505b-45e9-9788-cbe4b557917e","Type":"ContainerDied","Data":"94d29bf03388cbe8d79beb93341418e8ccfd1d6bfeb09089dec603d19dbff25b"} Oct 31 19:19:00.933276 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:00.933256 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" event={"ID":"81405623-de29-4fc6-bccf-447967669d89","Type":"ContainerStarted","Data":"eda1b17689a030351c8a85857a648f1c07d57f52704c3d9874708df34020ea9f"} Oct 31 19:19:00.933348 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:00.933310 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" event={"ID":"81405623-de29-4fc6-bccf-447967669d89","Type":"ContainerStarted","Data":"d22e04a6edfb1613986235781a2df1db1af11d1429f787e6a72f922eda9d882c"} Oct 31 19:19:00.933348 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:00.933342 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" event={"ID":"81405623-de29-4fc6-bccf-447967669d89","Type":"ContainerStarted","Data":"a3187af97ea8108d2e05fea82255f1e9c99697a868108a57c7b4fe6fa4041dc2"} Oct 31 19:19:01.442098 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:01.442013 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88a1fa4b-1191-434a-8e7c-783163ece24d-metrics-certs\") pod \"network-metrics-daemon-dfbrp\" (UID: \"88a1fa4b-1191-434a-8e7c-783163ece24d\") " pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:19:01.442098 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:01.442082 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9255a1d5-2514-441e-b942-6c39843e38f8-original-pull-secret\") pod \"global-pull-secret-syncer-lhtkb\" (UID: \"9255a1d5-2514-441e-b942-6c39843e38f8\") " pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:19:01.442265 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:01.442185 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 19:19:01.442265 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:01.442239 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9255a1d5-2514-441e-b942-6c39843e38f8-original-pull-secret podName:9255a1d5-2514-441e-b942-6c39843e38f8 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:17.442227038 +0000 UTC m=+34.256321247 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9255a1d5-2514-441e-b942-6c39843e38f8-original-pull-secret") pod "global-pull-secret-syncer-lhtkb" (UID: "9255a1d5-2514-441e-b942-6c39843e38f8") : object "kube-system"/"original-pull-secret" not registered Oct 31 19:19:01.442327 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:01.442185 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:19:01.442327 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:01.442306 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88a1fa4b-1191-434a-8e7c-783163ece24d-metrics-certs podName:88a1fa4b-1191-434a-8e7c-783163ece24d nodeName:}" failed. No retries permitted until 2026-10-31 19:19:17.44229474 +0000 UTC m=+34.256388962 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88a1fa4b-1191-434a-8e7c-783163ece24d-metrics-certs") pod "network-metrics-daemon-dfbrp" (UID: "88a1fa4b-1191-434a-8e7c-783163ece24d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 19:19:01.542796 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:01.542757 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7djgh\" (UniqueName: \"kubernetes.io/projected/23a10416-51b1-415f-b2fc-892b4d662b1f-kube-api-access-7djgh\") pod \"network-check-target-5mkxz\" (UID: \"23a10416-51b1-415f-b2fc-892b4d662b1f\") " pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:19:01.542991 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:01.542931 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 19:19:01.542991 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:01.542955 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 19:19:01.542991 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:01.542965 2566 projected.go:196] Error preparing data for projected volume kube-api-access-7djgh for pod openshift-network-diagnostics/network-check-target-5mkxz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:19:01.543101 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:01.543014 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23a10416-51b1-415f-b2fc-892b4d662b1f-kube-api-access-7djgh podName:23a10416-51b1-415f-b2fc-892b4d662b1f nodeName:}" failed. No retries permitted until 2026-10-31 19:19:17.543000859 +0000 UTC m=+34.357095071 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7djgh" (UniqueName: "kubernetes.io/projected/23a10416-51b1-415f-b2fc-892b4d662b1f-kube-api-access-7djgh") pod "network-check-target-5mkxz" (UID: "23a10416-51b1-415f-b2fc-892b4d662b1f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 19:19:01.731176 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:01.731097 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:19:01.731317 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:01.731270 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfbrp" podUID="88a1fa4b-1191-434a-8e7c-783163ece24d" Oct 31 19:19:01.731317 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:01.731313 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:19:01.731439 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:01.731420 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5mkxz" podUID="23a10416-51b1-415f-b2fc-892b4d662b1f" Oct 31 19:19:01.938310 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:01.938279 2566 generic.go:356] "Generic (PLEG): container finished" podID="47cff5a0-505b-45e9-9788-cbe4b557917e" containerID="1364f09091e32aec947d72de9eed8b75eb4f5ab25cc5abaf4999a1c7f58bd7e3" exitCode=0 Oct 31 19:19:01.938839 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:01.938356 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bhqht" event={"ID":"47cff5a0-505b-45e9-9788-cbe4b557917e","Type":"ContainerDied","Data":"1364f09091e32aec947d72de9eed8b75eb4f5ab25cc5abaf4999a1c7f58bd7e3"} Oct 31 19:19:01.941406 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:01.941385 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" event={"ID":"81405623-de29-4fc6-bccf-447967669d89","Type":"ContainerStarted","Data":"c96442e7355e7250ce1a420c5b9959896c245775faf54347d419b3e251e62692"} Oct 31 19:19:01.941470 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:01.941448 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" event={"ID":"81405623-de29-4fc6-bccf-447967669d89","Type":"ContainerStarted","Data":"a85a57a84331db7d5af24bee447451813faa4aacc72d63aa1ce060238aa12b74"} Oct 31 19:19:01.941514 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:01.941479 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" event={"ID":"81405623-de29-4fc6-bccf-447967669d89","Type":"ContainerStarted","Data":"311dad7fcef989ab5f8a203b966886c0b911445940980226c8c4b8690a9d7757"} Oct 31 19:19:02.731388 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:02.731359 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:19:02.731540 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:02.731454 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtkb" podUID="9255a1d5-2514-441e-b942-6c39843e38f8" Oct 31 19:19:02.946681 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:02.946644 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bhqht" event={"ID":"47cff5a0-505b-45e9-9788-cbe4b557917e","Type":"ContainerStarted","Data":"49c5cf3c20d23d5ac4d5bd286135b3a81f147b1f7c771edb342c2c7ea4dea5b5"} Oct 31 19:19:02.978604 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:02.978562 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bhqht" podStartSLOduration=8.106570931 podStartE2EDuration="18.978549135s" totalImagesPullingTime="10.871978204s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:02.976700865 +0000 UTC m=+19.790795093" watchObservedRunningTime="2026-10-31 19:19:02.978549135 +0000 UTC m=+19.792643366" Oct 31 19:19:03.732209 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:03.732182 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:19:03.732337 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:03.732273 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfbrp" podUID="88a1fa4b-1191-434a-8e7c-783163ece24d" Oct 31 19:19:03.732337 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:03.732325 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:19:03.732424 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:03.732405 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5mkxz" podUID="23a10416-51b1-415f-b2fc-892b4d662b1f" Oct 31 19:19:03.951868 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:03.951832 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" event={"ID":"81405623-de29-4fc6-bccf-447967669d89","Type":"ContainerStarted","Data":"a0f65460814750ff1d604609d3d32b1abab2364f7bedc3996394c6d4ece8d30d"} Oct 31 19:19:04.731045 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:04.731007 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:19:04.731216 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:04.731110 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtkb" podUID="9255a1d5-2514-441e-b942-6c39843e38f8" Oct 31 19:19:05.731680 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:05.731455 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:19:05.732206 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:05.731549 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:19:05.732206 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:05.731712 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5mkxz" podUID="23a10416-51b1-415f-b2fc-892b4d662b1f" Oct 31 19:19:05.732206 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:05.731822 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfbrp" podUID="88a1fa4b-1191-434a-8e7c-783163ece24d" Oct 31 19:19:05.958893 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:05.958858 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" event={"ID":"81405623-de29-4fc6-bccf-447967669d89","Type":"ContainerStarted","Data":"592cb8bd2a5415a8a79425091622e7b9cbbe2d8390a658b64ded87a112732eb1"} Oct 31 19:19:05.959215 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:05.959189 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:05.974810 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:05.974785 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:05.996768 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:05.996678 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" podStartSLOduration=7.22722707 podStartE2EDuration="21.996666587s" totalImagesPullingTime="14.769439517s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:05.996369806 +0000 UTC m=+22.810464038" watchObservedRunningTime="2026-10-31 19:19:05.996666587 +0000 UTC m=+22.810760818" Oct 31 19:19:06.731134 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:06.731103 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:19:06.731283 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:06.731209 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtkb" podUID="9255a1d5-2514-441e-b942-6c39843e38f8" Oct 31 19:19:06.961863 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:06.961834 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:06.962329 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:06.961952 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:06.978691 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:06.978667 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:07.007953 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:07.007868 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5mkxz"] Oct 31 19:19:07.008088 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:07.008006 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:19:07.008142 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:07.008104 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5mkxz" podUID="23a10416-51b1-415f-b2fc-892b4d662b1f" Oct 31 19:19:07.010724 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:07.010700 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lhtkb"] Oct 31 19:19:07.010859 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:07.010785 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:19:07.010935 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:07.010900 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtkb" podUID="9255a1d5-2514-441e-b942-6c39843e38f8" Oct 31 19:19:07.011354 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:07.011331 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dfbrp"] Oct 31 19:19:07.011454 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:07.011439 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:19:07.011565 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:07.011535 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfbrp" podUID="88a1fa4b-1191-434a-8e7c-783163ece24d" Oct 31 19:19:08.730849 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:08.730816 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:19:08.731345 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:08.730932 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:19:08.731345 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:08.730945 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtkb" podUID="9255a1d5-2514-441e-b942-6c39843e38f8" Oct 31 19:19:08.731345 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:08.730951 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:19:08.731345 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:08.731025 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfbrp" podUID="88a1fa4b-1191-434a-8e7c-783163ece24d" Oct 31 19:19:08.731345 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:08.731082 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5mkxz" podUID="23a10416-51b1-415f-b2fc-892b4d662b1f" Oct 31 19:19:10.730829 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:10.730797 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:19:10.730829 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:10.730820 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:19:10.731281 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:10.730899 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:19:10.731281 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:10.730925 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtkb" podUID="9255a1d5-2514-441e-b942-6c39843e38f8" Oct 31 19:19:10.731281 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:10.730993 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfbrp" podUID="88a1fa4b-1191-434a-8e7c-783163ece24d" Oct 31 19:19:10.731281 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:10.731073 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5mkxz" podUID="23a10416-51b1-415f-b2fc-892b4d662b1f" Oct 31 19:19:12.701091 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:12.701055 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wxdtf" Oct 31 19:19:12.701528 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:12.701191 2566 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 19:19:12.701741 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:12.701723 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wxdtf" Oct 31 19:19:12.731421 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:12.731393 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:19:12.731561 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:12.731484 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:19:12.731561 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:12.731497 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtkb" podUID="9255a1d5-2514-441e-b942-6c39843e38f8" Oct 31 19:19:12.731676 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:12.731563 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dfbrp" podUID="88a1fa4b-1191-434a-8e7c-783163ece24d" Oct 31 19:19:12.731676 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:12.731584 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:19:12.731765 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:12.731678 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5mkxz" podUID="23a10416-51b1-415f-b2fc-892b4d662b1f" Oct 31 19:19:13.467658 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.467496 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeReady" Oct 31 19:19:13.467815 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.467766 2566 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 19:19:13.507451 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.507417 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vszkf"] Oct 31 19:19:13.510417 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.510402 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-vszkf" Oct 31 19:19:13.514894 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.514867 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xmfjb"] Oct 31 19:19:13.517227 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.517211 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 19:19:13.517901 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.517880 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds"] Oct 31 19:19:13.518141 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.518120 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xmfjb" Oct 31 19:19:13.518795 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.518528 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 19:19:13.521780 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.521759 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4mdb2"] Oct 31 19:19:13.522139 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.521905 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" Oct 31 19:19:13.524639 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.524622 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c8ccd4bc6-xlml9"] Oct 31 19:19:13.524761 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.524746 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4mdb2" Oct 31 19:19:13.527376 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.527360 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-28mzh"] Oct 31 19:19:13.527503 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.527487 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:13.530016 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.529998 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-mtmp6"] Oct 31 19:19:13.530120 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.530105 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-28mzh" Oct 31 19:19:13.532518 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.532501 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4"] Oct 31 19:19:13.532627 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.532613 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:13.535539 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.535522 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgrs8"] Oct 31 19:19:13.535612 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.535602 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:13.538469 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.538453 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-8c75d8d7f-tzzdl"] Oct 31 19:19:13.538578 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.538563 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgrs8" Oct 31 19:19:13.541390 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.541372 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm"] Oct 31 19:19:13.541506 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.541493 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:13.541895 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.541876 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 19:19:13.544230 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.544209 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f878896cc-hzn9v"] Oct 31 19:19:13.544704 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.544608 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm" Oct 31 19:19:13.547056 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.547040 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-6vpwc"] Oct 31 19:19:13.547176 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.547158 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:13.552232 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.552209 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n"] Oct 31 19:19:13.552351 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.552332 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:13.554985 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.554968 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-l84pf"] Oct 31 19:19:13.555205 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.555190 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n" Oct 31 19:19:13.557903 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.557890 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vszkf"] Oct 31 19:19:13.557991 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.557971 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l84pf" Oct 31 19:19:13.559577 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.559558 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds"] Oct 31 19:19:13.559721 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.559692 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 19:19:13.566151 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.566128 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xmfjb"] Oct 31 19:19:13.573722 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.573704 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4mdb2"] Oct 31 19:19:13.574301 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.574285 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-mtmp6"] Oct 31 19:19:13.576029 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.576010 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-6vpwc"] Oct 31 19:19:13.576844 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.576812 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8c75d8d7f-tzzdl"] Oct 31 19:19:13.577428 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.577405 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n"] Oct 31 19:19:13.581063 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.581044 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-28mzh"] Oct 31 19:19:13.585802 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.585780 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f878896cc-hzn9v"] Oct 31 19:19:13.592004 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.591954 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4"] Oct 31 19:19:13.615978 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.615716 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-q8j58"] Oct 31 19:19:13.618597 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.618580 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q8j58" Oct 31 19:19:13.619280 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.619257 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 19:19:13.619372 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.619319 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 19:19:13.619372 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.619326 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 19:19:13.619372 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.619336 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 19:19:13.619553 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.619533 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 19:19:13.620317 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.620298 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 19:19:13.620392 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.620300 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 19:19:13.620392 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.620304 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 19:19:13.620392 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.620340 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 19:19:13.620392 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.620353 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 19:19:13.620756 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.620739 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 19:19:13.620835 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.620754 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 19:19:13.620835 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.620806 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 19:19:13.621156 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.621142 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5nm2g"] Oct 31 19:19:13.624116 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.624097 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:13.624262 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.624240 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxpcv\" (UniqueName: \"kubernetes.io/projected/4cd15118-0a59-4122-bdf1-56710c463262-kube-api-access-nxpcv\") pod \"cluster-monitoring-operator-884576c57-d4kds\" (UID: \"4cd15118-0a59-4122-bdf1-56710c463262\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" Oct 31 19:19:13.624332 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.624272 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5trh\" (UniqueName: \"kubernetes.io/projected/3a7f7d6d-958c-4973-9289-97f45526a431-kube-api-access-q5trh\") pod \"kas-connection-checker-685895c7c9-4mdb2\" (UID: \"3a7f7d6d-958c-4973-9289-97f45526a431\") " pod="kube-system/kas-connection-checker-685895c7c9-4mdb2" Oct 31 19:19:13.624332 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.624291 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a063ed7-8ce0-414e-8bbf-7733c464b507-installation-pull-secrets\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:13.624433 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.624343 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:13.624433 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.624366 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76e07820-d888-4138-a41a-59ff4565accd-config\") pod \"console-operator-7cd56b8dbc-6vpwc\" (UID: \"76e07820-d888-4138-a41a-59ff4565accd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:13.624433 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.624383 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76e07820-d888-4138-a41a-59ff4565accd-trusted-ca\") pod \"console-operator-7cd56b8dbc-6vpwc\" (UID: \"76e07820-d888-4138-a41a-59ff4565accd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:13.624591 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.624444 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/599fc0e5-05d8-46b0-90b3-60818a371161-service-ca-bundle\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:13.624591 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.624503 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zkn5\" (UniqueName: \"kubernetes.io/projected/ffb7495b-ef69-455c-99aa-22389d8c6d6e-kube-api-access-9zkn5\") pod \"volume-data-source-validator-58c9f5f46d-xgrs8\" (UID: \"ffb7495b-ef69-455c-99aa-22389d8c6d6e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgrs8" Oct 31 19:19:13.624591 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.624539 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-d4kds\" (UID: \"4cd15118-0a59-4122-bdf1-56710c463262\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" Oct 31 19:19:13.624591 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.624573 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-default-certificate\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:13.624744 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.624633 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cj8g\" (UniqueName: \"kubernetes.io/projected/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-kube-api-access-4cj8g\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:13.624744 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.624697 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:13.624744 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.624721 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-stats-auth\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:13.624838 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.624754 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-certificates\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:13.624838 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.624816 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/599fc0e5-05d8-46b0-90b3-60818a371161-tmp\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:13.624979 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.624842 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4cd15118-0a59-4122-bdf1-56710c463262-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-d4kds\" (UID: \"4cd15118-0a59-4122-bdf1-56710c463262\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" Oct 31 19:19:13.624979 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.624858 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ce261ba-08ff-418a-b3b4-b7e25e701bd8-serving-cert\") pod \"service-ca-operator-544d749c8c-kts6n\" (UID: \"4ce261ba-08ff-418a-b3b4-b7e25e701bd8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n" Oct 31 19:19:13.624979 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.624874 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb1a9bea-399b-4281-9ada-a232790cc713-ca-trust-extracted\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:13.624979 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.624961 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9q7w4\" (UID: \"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:13.625116 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625042 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqspp\" (UniqueName: \"kubernetes.io/projected/76e07820-d888-4138-a41a-59ff4565accd-kube-api-access-dqspp\") pod \"console-operator-7cd56b8dbc-6vpwc\" (UID: \"76e07820-d888-4138-a41a-59ff4565accd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:13.625116 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625073 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qtrj\" (UniqueName: \"kubernetes.io/projected/4ce261ba-08ff-418a-b3b4-b7e25e701bd8-kube-api-access-4qtrj\") pod \"service-ca-operator-544d749c8c-kts6n\" (UID: \"4ce261ba-08ff-418a-b3b4-b7e25e701bd8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n" Oct 31 19:19:13.625116 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625095 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4wnr\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-kube-api-access-t4wnr\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:13.625215 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625138 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a063ed7-8ce0-414e-8bbf-7733c464b507-ca-trust-extracted\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:13.625215 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625168 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a063ed7-8ce0-414e-8bbf-7733c464b507-image-registry-private-configuration\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:13.625215 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625190 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-bound-sa-token\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:13.625215 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625205 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/599fc0e5-05d8-46b0-90b3-60818a371161-serving-cert\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:13.625354 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625234 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4lhh\" (UniqueName: \"kubernetes.io/projected/599fc0e5-05d8-46b0-90b3-60818a371161-kube-api-access-b4lhh\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:13.625354 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625261 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ce261ba-08ff-418a-b3b4-b7e25e701bd8-config\") pod \"service-ca-operator-544d749c8c-kts6n\" (UID: \"4ce261ba-08ff-418a-b3b4-b7e25e701bd8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n" Oct 31 19:19:13.625354 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625280 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-bound-sa-token\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:13.625354 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625315 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-58hlm\" (UID: \"ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm" Oct 31 19:19:13.625549 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625363 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb1a9bea-399b-4281-9ada-a232790cc713-installation-pull-secrets\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:13.625549 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625400 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g49vm\" (UniqueName: \"kubernetes.io/projected/3e84edf8-6e35-489b-b19b-17a5179e666e-kube-api-access-g49vm\") pod \"network-check-source-6f78d59f99-l84pf\" (UID: \"3e84edf8-6e35-489b-b19b-17a5179e666e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l84pf" Oct 31 19:19:13.625549 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625436 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp54m\" (UniqueName: \"kubernetes.io/projected/ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f-kube-api-access-gp54m\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-58hlm\" (UID: \"ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm" Oct 31 19:19:13.625549 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625473 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/599fc0e5-05d8-46b0-90b3-60818a371161-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:13.625549 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625523 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-nginx-conf\") pod \"networking-console-plugin-86c478bb88-28mzh\" (UID: \"9cf7d7f5-3936-45b3-ba2c-597cf99cfb89\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-28mzh" Oct 31 19:19:13.625787 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625557 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-58hlm\" (UID: \"ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm" Oct 31 19:19:13.625787 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625620 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mk7v\" (UniqueName: \"kubernetes.io/projected/211efbea-29d0-4b2f-8c9e-6f3badcf77e5-kube-api-access-9mk7v\") pod \"kas-connection-checker-685895c7c9-vszkf\" (UID: \"211efbea-29d0-4b2f-8c9e-6f3badcf77e5\") " pod="kube-system/kas-connection-checker-685895c7c9-vszkf" Oct 31 19:19:13.625787 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625682 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb1a9bea-399b-4281-9ada-a232790cc713-registry-certificates\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:13.625787 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625706 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-9q7w4\" (UID: \"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:13.625787 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625742 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zsbj\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-kube-api-access-4zsbj\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:13.625787 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625761 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb1a9bea-399b-4281-9ada-a232790cc713-trusted-ca\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:13.625787 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625776 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-28mzh\" (UID: \"9cf7d7f5-3936-45b3-ba2c-597cf99cfb89\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-28mzh" Oct 31 19:19:13.626215 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625807 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:13.626215 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625829 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6wxr\" (UniqueName: \"kubernetes.io/projected/1cc2220b-f30e-4879-b2cb-b2dd8e53b9f0-kube-api-access-z6wxr\") pod \"kas-connection-checker-685895c7c9-xmfjb\" (UID: \"1cc2220b-f30e-4879-b2cb-b2dd8e53b9f0\") " pod="kube-system/kas-connection-checker-685895c7c9-xmfjb" Oct 31 19:19:13.626215 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625863 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76e07820-d888-4138-a41a-59ff4565accd-serving-cert\") pod \"console-operator-7cd56b8dbc-6vpwc\" (UID: \"76e07820-d888-4138-a41a-59ff4565accd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:13.626215 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625884 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/599fc0e5-05d8-46b0-90b3-60818a371161-snapshots\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:13.626215 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625905 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:13.626215 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625950 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w48sj\" (UniqueName: \"kubernetes.io/projected/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-kube-api-access-w48sj\") pod \"cluster-samples-operator-6ddb6c8b94-9q7w4\" (UID: \"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:13.626215 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.625984 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb1a9bea-399b-4281-9ada-a232790cc713-image-registry-private-configuration\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:13.626215 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.626072 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a063ed7-8ce0-414e-8bbf-7733c464b507-trusted-ca\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:13.629515 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.629493 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 19:19:13.629619 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.629537 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 19:19:13.629619 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.629545 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 19:19:13.629619 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.629551 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 19:19:13.629619 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.629570 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 19:19:13.631529 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.631504 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 19:19:13.631616 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.631571 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 19:19:13.631677 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.631636 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 19:19:13.631677 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.631641 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 19:19:13.631778 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.631689 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 19:19:13.632870 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.632849 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 19:19:13.634499 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.634478 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 19:19:13.644891 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.640972 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgrs8"] Oct 31 19:19:13.644891 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.642567 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 19:19:13.649977 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.649958 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-l84pf"] Oct 31 19:19:13.651015 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.650993 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c8ccd4bc6-xlml9"] Oct 31 19:19:13.653105 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.653083 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 19:19:13.653105 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.653083 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 19:19:13.653251 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.653115 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 19:19:13.653251 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.653168 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 19:19:13.653970 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.653952 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5nm2g"] Oct 31 19:19:13.654324 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.654308 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 19:19:13.654749 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.654731 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 19:19:13.654824 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.654760 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 19:19:13.654824 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.654772 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 19:19:13.654824 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.654775 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 19:19:13.654824 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.654739 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 19:19:13.654824 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.654824 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 19:19:13.655026 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.654834 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 19:19:13.655026 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.654857 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 19:19:13.656443 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.656346 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 19:19:13.656443 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.656376 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vq9kc\"" Oct 31 19:19:13.656443 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.656407 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q7xwb\"" Oct 31 19:19:13.656443 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.656409 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 19:19:13.656629 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.656467 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mrxc5\"" Oct 31 19:19:13.657028 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.656980 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 19:19:13.658223 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.658206 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q8j58"] Oct 31 19:19:13.658762 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.658747 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm"] Oct 31 19:19:13.716728 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.716699 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 19:19:13.716728 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.716699 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 19:19:13.717174 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.716842 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 19:19:13.726731 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.726703 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/599fc0e5-05d8-46b0-90b3-60818a371161-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:13.726731 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.726729 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-nginx-conf\") pod \"networking-console-plugin-86c478bb88-28mzh\" (UID: \"9cf7d7f5-3936-45b3-ba2c-597cf99cfb89\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-28mzh" Oct 31 19:19:13.726962 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.726744 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-58hlm\" (UID: \"ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm" Oct 31 19:19:13.726962 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.726766 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mk7v\" (UniqueName: \"kubernetes.io/projected/211efbea-29d0-4b2f-8c9e-6f3badcf77e5-kube-api-access-9mk7v\") pod \"kas-connection-checker-685895c7c9-vszkf\" (UID: \"211efbea-29d0-4b2f-8c9e-6f3badcf77e5\") " pod="kube-system/kas-connection-checker-685895c7c9-vszkf" Oct 31 19:19:13.726962 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.726947 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb1a9bea-399b-4281-9ada-a232790cc713-registry-certificates\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:13.727128 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.726980 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-9q7w4\" (UID: \"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:13.727128 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727001 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zsbj\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-kube-api-access-4zsbj\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:13.727128 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727017 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb1a9bea-399b-4281-9ada-a232790cc713-trusted-ca\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:13.727128 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727041 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-28mzh\" (UID: \"9cf7d7f5-3936-45b3-ba2c-597cf99cfb89\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-28mzh" Oct 31 19:19:13.727128 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727070 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:13.727128 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727094 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6wxr\" (UniqueName: \"kubernetes.io/projected/1cc2220b-f30e-4879-b2cb-b2dd8e53b9f0-kube-api-access-z6wxr\") pod \"kas-connection-checker-685895c7c9-xmfjb\" (UID: \"1cc2220b-f30e-4879-b2cb-b2dd8e53b9f0\") " pod="kube-system/kas-connection-checker-685895c7c9-xmfjb" Oct 31 19:19:13.727128 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727123 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76e07820-d888-4138-a41a-59ff4565accd-serving-cert\") pod \"console-operator-7cd56b8dbc-6vpwc\" (UID: \"76e07820-d888-4138-a41a-59ff4565accd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:13.727443 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727152 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/599fc0e5-05d8-46b0-90b3-60818a371161-snapshots\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:13.727443 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727175 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:13.727443 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727197 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w48sj\" (UniqueName: \"kubernetes.io/projected/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-kube-api-access-w48sj\") pod \"cluster-samples-operator-6ddb6c8b94-9q7w4\" (UID: \"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:13.727443 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727226 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb1a9bea-399b-4281-9ada-a232790cc713-image-registry-private-configuration\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:13.727443 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727266 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a063ed7-8ce0-414e-8bbf-7733c464b507-trusted-ca\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:13.727443 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727301 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpcv\" (UniqueName: \"kubernetes.io/projected/4cd15118-0a59-4122-bdf1-56710c463262-kube-api-access-nxpcv\") pod \"cluster-monitoring-operator-884576c57-d4kds\" (UID: \"4cd15118-0a59-4122-bdf1-56710c463262\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" Oct 31 19:19:13.727443 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727331 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5418e7b8-1471-4aa5-ad70-5d60b87c572a-tmp-dir\") pod \"dns-default-5nm2g\" (UID: \"5418e7b8-1471-4aa5-ad70-5d60b87c572a\") " pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:13.727443 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727355 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92aaddbc-6268-443c-b3bd-9f74ab6b9a4b-cert\") pod \"ingress-canary-q8j58\" (UID: \"92aaddbc-6268-443c-b3bd-9f74ab6b9a4b\") " pod="openshift-ingress-canary/ingress-canary-q8j58" Oct 31 19:19:13.727443 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727382 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5trh\" (UniqueName: \"kubernetes.io/projected/3a7f7d6d-958c-4973-9289-97f45526a431-kube-api-access-q5trh\") pod \"kas-connection-checker-685895c7c9-4mdb2\" (UID: \"3a7f7d6d-958c-4973-9289-97f45526a431\") " pod="kube-system/kas-connection-checker-685895c7c9-4mdb2" Oct 31 19:19:13.727443 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727410 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a063ed7-8ce0-414e-8bbf-7733c464b507-installation-pull-secrets\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:13.727443 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727438 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzwvn\" (UniqueName: \"kubernetes.io/projected/5418e7b8-1471-4aa5-ad70-5d60b87c572a-kube-api-access-vzwvn\") pod \"dns-default-5nm2g\" (UID: \"5418e7b8-1471-4aa5-ad70-5d60b87c572a\") " pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:13.728021 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727462 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:13.728021 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727487 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76e07820-d888-4138-a41a-59ff4565accd-config\") pod \"console-operator-7cd56b8dbc-6vpwc\" (UID: \"76e07820-d888-4138-a41a-59ff4565accd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:13.728021 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727511 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76e07820-d888-4138-a41a-59ff4565accd-trusted-ca\") pod \"console-operator-7cd56b8dbc-6vpwc\" (UID: \"76e07820-d888-4138-a41a-59ff4565accd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:13.728021 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727542 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/599fc0e5-05d8-46b0-90b3-60818a371161-service-ca-bundle\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:13.728021 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727582 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-nginx-conf\") pod \"networking-console-plugin-86c478bb88-28mzh\" (UID: \"9cf7d7f5-3936-45b3-ba2c-597cf99cfb89\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-28mzh" Oct 31 19:19:13.728021 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727589 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zkn5\" (UniqueName: \"kubernetes.io/projected/ffb7495b-ef69-455c-99aa-22389d8c6d6e-kube-api-access-9zkn5\") pod \"volume-data-source-validator-58c9f5f46d-xgrs8\" (UID: \"ffb7495b-ef69-455c-99aa-22389d8c6d6e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgrs8" Oct 31 19:19:13.728021 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727635 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5418e7b8-1471-4aa5-ad70-5d60b87c572a-config-volume\") pod \"dns-default-5nm2g\" (UID: \"5418e7b8-1471-4aa5-ad70-5d60b87c572a\") " pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:13.728021 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727672 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-d4kds\" (UID: \"4cd15118-0a59-4122-bdf1-56710c463262\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" Oct 31 19:19:13.728021 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727700 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-default-certificate\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:13.728021 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727726 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cj8g\" (UniqueName: \"kubernetes.io/projected/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-kube-api-access-4cj8g\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:13.728021 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:13.727796 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 19:19:13.728021 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727801 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:13.728021 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727830 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-stats-auth\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:13.728021 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:13.727850 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls podName:4cd15118-0a59-4122-bdf1-56710c463262 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:14.227833645 +0000 UTC m=+31.041927855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-d4kds" (UID: "4cd15118-0a59-4122-bdf1-56710c463262") : secret "cluster-monitoring-operator-tls" not found Oct 31 19:19:13.728021 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:13.727881 2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 19:19:13.728021 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727885 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-certificates\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:13.728852 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727893 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-9q7w4\" (UID: \"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:13.728852 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727929 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/599fc0e5-05d8-46b0-90b3-60818a371161-tmp\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:13.728852 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:13.727948 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert podName:9cf7d7f5-3936-45b3-ba2c-597cf99cfb89 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:14.227933454 +0000 UTC m=+31.042027679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-28mzh" (UID: "9cf7d7f5-3936-45b3-ba2c-597cf99cfb89") : secret "networking-console-plugin-cert" not found Oct 31 19:19:13.728852 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.727879 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb1a9bea-399b-4281-9ada-a232790cc713-registry-certificates\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:13.728852 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.728036 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/599fc0e5-05d8-46b0-90b3-60818a371161-service-ca-bundle\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:13.728852 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.728116 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4cd15118-0a59-4122-bdf1-56710c463262-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-d4kds\" (UID: \"4cd15118-0a59-4122-bdf1-56710c463262\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" Oct 31 19:19:13.728852 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.728147 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ce261ba-08ff-418a-b3b4-b7e25e701bd8-serving-cert\") pod \"service-ca-operator-544d749c8c-kts6n\" (UID: \"4ce261ba-08ff-418a-b3b4-b7e25e701bd8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n" Oct 31 19:19:13.728852 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.728181 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb1a9bea-399b-4281-9ada-a232790cc713-ca-trust-extracted\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:13.728852 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.728225 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9q7w4\" (UID: \"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:13.728852 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.728242 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/599fc0e5-05d8-46b0-90b3-60818a371161-tmp\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:13.728852 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.728265 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqspp\" (UniqueName: \"kubernetes.io/projected/76e07820-d888-4138-a41a-59ff4565accd-kube-api-access-dqspp\") pod \"console-operator-7cd56b8dbc-6vpwc\" (UID: \"76e07820-d888-4138-a41a-59ff4565accd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:13.728852 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.728296 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qtrj\" (UniqueName: \"kubernetes.io/projected/4ce261ba-08ff-418a-b3b4-b7e25e701bd8-kube-api-access-4qtrj\") pod \"service-ca-operator-544d749c8c-kts6n\" (UID: \"4ce261ba-08ff-418a-b3b4-b7e25e701bd8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n" Oct 31 19:19:13.728852 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.728323 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4wnr\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-kube-api-access-t4wnr\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:13.728852 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.728352 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxnqt\" (UniqueName: \"kubernetes.io/projected/92aaddbc-6268-443c-b3bd-9f74ab6b9a4b-kube-api-access-rxnqt\") pod \"ingress-canary-q8j58\" (UID: \"92aaddbc-6268-443c-b3bd-9f74ab6b9a4b\") " pod="openshift-ingress-canary/ingress-canary-q8j58" Oct 31 19:19:13.728852 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.728470 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a063ed7-8ce0-414e-8bbf-7733c464b507-ca-trust-extracted\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:13.729511 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.728501 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a063ed7-8ce0-414e-8bbf-7733c464b507-image-registry-private-configuration\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:13.729511 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.728527 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-bound-sa-token\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:13.729511 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.728550 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/599fc0e5-05d8-46b0-90b3-60818a371161-serving-cert\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:13.729511 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.728574 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4lhh\" (UniqueName: \"kubernetes.io/projected/599fc0e5-05d8-46b0-90b3-60818a371161-kube-api-access-b4lhh\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:13.729511 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.728607 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ce261ba-08ff-418a-b3b4-b7e25e701bd8-config\") pod \"service-ca-operator-544d749c8c-kts6n\" (UID: \"4ce261ba-08ff-418a-b3b4-b7e25e701bd8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n" Oct 31 19:19:13.729511 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.728631 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-bound-sa-token\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:13.729511 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.728656 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-58hlm\" (UID: \"ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm" Oct 31 19:19:13.729511 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.728665 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-certificates\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:13.729511 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.728681 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5418e7b8-1471-4aa5-ad70-5d60b87c572a-metrics-tls\") pod \"dns-default-5nm2g\" (UID: \"5418e7b8-1471-4aa5-ad70-5d60b87c572a\") " pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:13.729511 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.728707 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb1a9bea-399b-4281-9ada-a232790cc713-installation-pull-secrets\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:13.729511 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.728731 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g49vm\" (UniqueName: \"kubernetes.io/projected/3e84edf8-6e35-489b-b19b-17a5179e666e-kube-api-access-g49vm\") pod \"network-check-source-6f78d59f99-l84pf\" (UID: \"3e84edf8-6e35-489b-b19b-17a5179e666e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l84pf" Oct 31 19:19:13.729511 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.728756 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp54m\" (UniqueName: \"kubernetes.io/projected/ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f-kube-api-access-gp54m\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-58hlm\" (UID: \"ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm" Oct 31 19:19:13.729511 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.729009 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb1a9bea-399b-4281-9ada-a232790cc713-trusted-ca\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:13.729511 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.729115 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/599fc0e5-05d8-46b0-90b3-60818a371161-snapshots\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:13.729511 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.729182 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4cd15118-0a59-4122-bdf1-56710c463262-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-d4kds\" (UID: \"4cd15118-0a59-4122-bdf1-56710c463262\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" Oct 31 19:19:13.729511 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:13.729192 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 19:19:13.730282 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:13.729239 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs podName:70c870bc-c9e6-4d7e-bc99-ba556ce31cde nodeName:}" failed. No retries permitted until 2026-10-31 19:19:14.22922614 +0000 UTC m=+31.043320350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs") pod "router-default-8c75d8d7f-tzzdl" (UID: "70c870bc-c9e6-4d7e-bc99-ba556ce31cde") : secret "router-metrics-certs-default" not found Oct 31 19:19:13.730282 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.729261 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a063ed7-8ce0-414e-8bbf-7733c464b507-trusted-ca\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:13.730282 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:13.729406 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 19:19:13.730282 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:13.729421 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8ccd4bc6-xlml9: secret "image-registry-tls" not found Oct 31 19:19:13.730282 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:13.729457 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls podName:3a063ed7-8ce0-414e-8bbf-7733c464b507 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:14.229444928 +0000 UTC m=+31.043539155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls") pod "image-registry-5c8ccd4bc6-xlml9" (UID: "3a063ed7-8ce0-414e-8bbf-7733c464b507") : secret "image-registry-tls" not found Oct 31 19:19:13.730282 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.729630 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb1a9bea-399b-4281-9ada-a232790cc713-ca-trust-extracted\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:13.730282 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:13.729707 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 19:19:13.730282 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:13.729753 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls podName:f4d13f0a-4f53-4659-9a4e-0e1020f98b2e nodeName:}" failed. No retries permitted until 2026-10-31 19:19:14.229740329 +0000 UTC m=+31.043834552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-9q7w4" (UID: "f4d13f0a-4f53-4659-9a4e-0e1020f98b2e") : secret "samples-operator-tls" not found Oct 31 19:19:13.730282 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:13.730027 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 19:19:13.730282 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:13.730044 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f878896cc-hzn9v: secret "image-registry-tls" not found Oct 31 19:19:13.730282 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:13.730098 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls podName:eb1a9bea-399b-4281-9ada-a232790cc713 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:14.230083579 +0000 UTC m=+31.044177816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls") pod "image-registry-6f878896cc-hzn9v" (UID: "eb1a9bea-399b-4281-9ada-a232790cc713") : secret "image-registry-tls" not found Oct 31 19:19:13.730282 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.730187 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76e07820-d888-4138-a41a-59ff4565accd-config\") pod \"console-operator-7cd56b8dbc-6vpwc\" (UID: \"76e07820-d888-4138-a41a-59ff4565accd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:13.730890 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.730600 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a063ed7-8ce0-414e-8bbf-7733c464b507-ca-trust-extracted\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:13.731157 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.731133 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76e07820-d888-4138-a41a-59ff4565accd-trusted-ca\") pod \"console-operator-7cd56b8dbc-6vpwc\" (UID: \"76e07820-d888-4138-a41a-59ff4565accd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:13.731666 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:13.731649 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle podName:70c870bc-c9e6-4d7e-bc99-ba556ce31cde nodeName:}" failed. No retries permitted until 2026-10-31 19:19:14.231630593 +0000 UTC m=+31.045724820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle") pod "router-default-8c75d8d7f-tzzdl" (UID: "70c870bc-c9e6-4d7e-bc99-ba556ce31cde") : configmap references non-existent config key: service-ca.crt Oct 31 19:19:13.732038 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.732007 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-58hlm\" (UID: \"ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm" Oct 31 19:19:13.732249 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.732230 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/599fc0e5-05d8-46b0-90b3-60818a371161-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:13.732594 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.732573 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ce261ba-08ff-418a-b3b4-b7e25e701bd8-config\") pod \"service-ca-operator-544d749c8c-kts6n\" (UID: \"4ce261ba-08ff-418a-b3b4-b7e25e701bd8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n" Oct 31 19:19:13.732665 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.732575 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-58hlm\" (UID: \"ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm" Oct 31 19:19:13.732829 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.732808 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-stats-auth\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:13.733037 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.733015 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a063ed7-8ce0-414e-8bbf-7733c464b507-installation-pull-secrets\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:13.733242 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.733220 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ce261ba-08ff-418a-b3b4-b7e25e701bd8-serving-cert\") pod \"service-ca-operator-544d749c8c-kts6n\" (UID: \"4ce261ba-08ff-418a-b3b4-b7e25e701bd8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n" Oct 31 19:19:13.733349 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.733328 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76e07820-d888-4138-a41a-59ff4565accd-serving-cert\") pod \"console-operator-7cd56b8dbc-6vpwc\" (UID: \"76e07820-d888-4138-a41a-59ff4565accd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:13.733926 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.733868 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-default-certificate\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:13.734043 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.734024 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/599fc0e5-05d8-46b0-90b3-60818a371161-serving-cert\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:13.734233 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.734217 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a063ed7-8ce0-414e-8bbf-7733c464b507-image-registry-private-configuration\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:13.734867 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.734847 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb1a9bea-399b-4281-9ada-a232790cc713-image-registry-private-configuration\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:13.735854 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.735832 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb1a9bea-399b-4281-9ada-a232790cc713-installation-pull-secrets\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:13.738450 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.738430 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mk7v\" (UniqueName: \"kubernetes.io/projected/211efbea-29d0-4b2f-8c9e-6f3badcf77e5-kube-api-access-9mk7v\") pod \"kas-connection-checker-685895c7c9-vszkf\" (UID: \"211efbea-29d0-4b2f-8c9e-6f3badcf77e5\") " pod="kube-system/kas-connection-checker-685895c7c9-vszkf" Oct 31 19:19:13.739026 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.739009 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zsbj\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-kube-api-access-4zsbj\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:13.742524 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.742503 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxpcv\" (UniqueName: \"kubernetes.io/projected/4cd15118-0a59-4122-bdf1-56710c463262-kube-api-access-nxpcv\") pod \"cluster-monitoring-operator-884576c57-d4kds\" (UID: \"4cd15118-0a59-4122-bdf1-56710c463262\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" Oct 31 19:19:13.742600 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.742584 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zkn5\" (UniqueName: \"kubernetes.io/projected/ffb7495b-ef69-455c-99aa-22389d8c6d6e-kube-api-access-9zkn5\") pod \"volume-data-source-validator-58c9f5f46d-xgrs8\" (UID: \"ffb7495b-ef69-455c-99aa-22389d8c6d6e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgrs8" Oct 31 19:19:13.746133 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.746110 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5trh\" (UniqueName: \"kubernetes.io/projected/3a7f7d6d-958c-4973-9289-97f45526a431-kube-api-access-q5trh\") pod \"kas-connection-checker-685895c7c9-4mdb2\" (UID: \"3a7f7d6d-958c-4973-9289-97f45526a431\") " pod="kube-system/kas-connection-checker-685895c7c9-4mdb2" Oct 31 19:19:13.746633 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.746605 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-bound-sa-token\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:13.748080 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.748054 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-bound-sa-token\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:13.748470 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.748449 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqspp\" (UniqueName: \"kubernetes.io/projected/76e07820-d888-4138-a41a-59ff4565accd-kube-api-access-dqspp\") pod \"console-operator-7cd56b8dbc-6vpwc\" (UID: \"76e07820-d888-4138-a41a-59ff4565accd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:13.752485 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.752456 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cj8g\" (UniqueName: \"kubernetes.io/projected/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-kube-api-access-4cj8g\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:13.753855 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.753829 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4lhh\" (UniqueName: \"kubernetes.io/projected/599fc0e5-05d8-46b0-90b3-60818a371161-kube-api-access-b4lhh\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:13.754490 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.754470 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g49vm\" (UniqueName: \"kubernetes.io/projected/3e84edf8-6e35-489b-b19b-17a5179e666e-kube-api-access-g49vm\") pod \"network-check-source-6f78d59f99-l84pf\" (UID: \"3e84edf8-6e35-489b-b19b-17a5179e666e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l84pf" Oct 31 19:19:13.755050 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.755032 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4wnr\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-kube-api-access-t4wnr\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:13.755342 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.755322 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp54m\" (UniqueName: \"kubernetes.io/projected/ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f-kube-api-access-gp54m\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-58hlm\" (UID: \"ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm" Oct 31 19:19:13.756483 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.756461 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qtrj\" (UniqueName: \"kubernetes.io/projected/4ce261ba-08ff-418a-b3b4-b7e25e701bd8-kube-api-access-4qtrj\") pod \"service-ca-operator-544d749c8c-kts6n\" (UID: \"4ce261ba-08ff-418a-b3b4-b7e25e701bd8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n" Oct 31 19:19:13.756632 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.756614 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w48sj\" (UniqueName: \"kubernetes.io/projected/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-kube-api-access-w48sj\") pod \"cluster-samples-operator-6ddb6c8b94-9q7w4\" (UID: \"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:13.757537 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.757518 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6wxr\" (UniqueName: \"kubernetes.io/projected/1cc2220b-f30e-4879-b2cb-b2dd8e53b9f0-kube-api-access-z6wxr\") pod \"kas-connection-checker-685895c7c9-xmfjb\" (UID: \"1cc2220b-f30e-4879-b2cb-b2dd8e53b9f0\") " pod="kube-system/kas-connection-checker-685895c7c9-xmfjb" Oct 31 19:19:13.824523 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.824495 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-vszkf" Oct 31 19:19:13.829105 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.829082 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzwvn\" (UniqueName: \"kubernetes.io/projected/5418e7b8-1471-4aa5-ad70-5d60b87c572a-kube-api-access-vzwvn\") pod \"dns-default-5nm2g\" (UID: \"5418e7b8-1471-4aa5-ad70-5d60b87c572a\") " pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:13.829172 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.829133 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5418e7b8-1471-4aa5-ad70-5d60b87c572a-config-volume\") pod \"dns-default-5nm2g\" (UID: \"5418e7b8-1471-4aa5-ad70-5d60b87c572a\") " pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:13.829243 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.829229 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxnqt\" (UniqueName: \"kubernetes.io/projected/92aaddbc-6268-443c-b3bd-9f74ab6b9a4b-kube-api-access-rxnqt\") pod \"ingress-canary-q8j58\" (UID: \"92aaddbc-6268-443c-b3bd-9f74ab6b9a4b\") " pod="openshift-ingress-canary/ingress-canary-q8j58" Oct 31 19:19:13.829282 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.829259 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5418e7b8-1471-4aa5-ad70-5d60b87c572a-metrics-tls\") pod \"dns-default-5nm2g\" (UID: \"5418e7b8-1471-4aa5-ad70-5d60b87c572a\") " pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:13.829356 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.829346 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5418e7b8-1471-4aa5-ad70-5d60b87c572a-tmp-dir\") pod \"dns-default-5nm2g\" (UID: \"5418e7b8-1471-4aa5-ad70-5d60b87c572a\") " pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:13.829392 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.829362 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92aaddbc-6268-443c-b3bd-9f74ab6b9a4b-cert\") pod \"ingress-canary-q8j58\" (UID: \"92aaddbc-6268-443c-b3bd-9f74ab6b9a4b\") " pod="openshift-ingress-canary/ingress-canary-q8j58" Oct 31 19:19:13.829782 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:13.829763 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 19:19:13.829925 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:13.829823 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5418e7b8-1471-4aa5-ad70-5d60b87c572a-metrics-tls podName:5418e7b8-1471-4aa5-ad70-5d60b87c572a nodeName:}" failed. No retries permitted until 2026-10-31 19:19:14.329806048 +0000 UTC m=+31.143900271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5418e7b8-1471-4aa5-ad70-5d60b87c572a-metrics-tls") pod "dns-default-5nm2g" (UID: "5418e7b8-1471-4aa5-ad70-5d60b87c572a") : secret "dns-default-metrics-tls" not found Oct 31 19:19:13.829925 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.829825 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5418e7b8-1471-4aa5-ad70-5d60b87c572a-config-volume\") pod \"dns-default-5nm2g\" (UID: \"5418e7b8-1471-4aa5-ad70-5d60b87c572a\") " pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:13.830068 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.830046 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5418e7b8-1471-4aa5-ad70-5d60b87c572a-tmp-dir\") pod \"dns-default-5nm2g\" (UID: \"5418e7b8-1471-4aa5-ad70-5d60b87c572a\") " pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:13.831780 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.831756 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92aaddbc-6268-443c-b3bd-9f74ab6b9a4b-cert\") pod \"ingress-canary-q8j58\" (UID: \"92aaddbc-6268-443c-b3bd-9f74ab6b9a4b\") " pod="openshift-ingress-canary/ingress-canary-q8j58" Oct 31 19:19:13.831862 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.831848 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xmfjb" Oct 31 19:19:13.841611 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.841582 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzwvn\" (UniqueName: \"kubernetes.io/projected/5418e7b8-1471-4aa5-ad70-5d60b87c572a-kube-api-access-vzwvn\") pod \"dns-default-5nm2g\" (UID: \"5418e7b8-1471-4aa5-ad70-5d60b87c572a\") " pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:13.843655 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.843613 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxnqt\" (UniqueName: \"kubernetes.io/projected/92aaddbc-6268-443c-b3bd-9f74ab6b9a4b-kube-api-access-rxnqt\") pod \"ingress-canary-q8j58\" (UID: \"92aaddbc-6268-443c-b3bd-9f74ab6b9a4b\") " pod="openshift-ingress-canary/ingress-canary-q8j58" Oct 31 19:19:13.845179 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.845162 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4mdb2" Oct 31 19:19:13.869943 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.869787 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:13.885868 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.885627 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgrs8" Oct 31 19:19:13.902595 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.902561 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm" Oct 31 19:19:13.921846 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.921815 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:13.933739 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.933597 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n" Oct 31 19:19:13.944006 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.939999 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l84pf" Oct 31 19:19:13.954563 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:13.953867 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q8j58" Oct 31 19:19:14.118211 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.118186 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vszkf"] Oct 31 19:19:14.120966 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.120709 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xmfjb"] Oct 31 19:19:14.120966 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:19:14.120787 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod211efbea_29d0_4b2f_8c9e_6f3badcf77e5.slice/crio-860bdd126bdb23ddbab4d48802a3be431f2d92a95f5d5716ba177928be8c2092 WatchSource:0}: Error finding container 860bdd126bdb23ddbab4d48802a3be431f2d92a95f5d5716ba177928be8c2092: Status 404 returned error can't find the container with id 860bdd126bdb23ddbab4d48802a3be431f2d92a95f5d5716ba177928be8c2092 Oct 31 19:19:14.122214 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:19:14.122194 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc2220b_f30e_4879_b2cb_b2dd8e53b9f0.slice/crio-336ffea7ff08409e19669798ffcc44ea0d354964196e08aafc954f9396d4ca05 WatchSource:0}: Error finding container 336ffea7ff08409e19669798ffcc44ea0d354964196e08aafc954f9396d4ca05: Status 404 returned error can't find the container with id 336ffea7ff08409e19669798ffcc44ea0d354964196e08aafc954f9396d4ca05 Oct 31 19:19:14.127443 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.127420 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4mdb2"] Oct 31 19:19:14.129560 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:19:14.129540 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7f7d6d_958c_4973_9289_97f45526a431.slice/crio-0a7f85425d882bdc757d28a51639f891ce7c284d793f8bfb07690ada43662c58 WatchSource:0}: Error finding container 0a7f85425d882bdc757d28a51639f891ce7c284d793f8bfb07690ada43662c58: Status 404 returned error can't find the container with id 0a7f85425d882bdc757d28a51639f891ce7c284d793f8bfb07690ada43662c58 Oct 31 19:19:14.200111 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.199989 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-mtmp6"] Oct 31 19:19:14.206004 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:19:14.205904 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod599fc0e5_05d8_46b0_90b3_60818a371161.slice/crio-ada53789e04a83feca4c30151511cf86a4a6d7df24944c1c83b863ced5a2136c WatchSource:0}: Error finding container ada53789e04a83feca4c30151511cf86a4a6d7df24944c1c83b863ced5a2136c: Status 404 returned error can't find the container with id ada53789e04a83feca4c30151511cf86a4a6d7df24944c1c83b863ced5a2136c Oct 31 19:19:14.216024 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.215996 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgrs8"] Oct 31 19:19:14.223603 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.223376 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm"] Oct 31 19:19:14.226012 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:19:14.225978 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8a8fb6_c472_4e79_90bb_8a96c61e3b2f.slice/crio-f3b2be88f679fe1cf5dd30b6b6873254fe93aa6dcff09dbb01538f980e3642bd WatchSource:0}: Error finding container f3b2be88f679fe1cf5dd30b6b6873254fe93aa6dcff09dbb01538f980e3642bd: Status 404 returned error can't find the container with id f3b2be88f679fe1cf5dd30b6b6873254fe93aa6dcff09dbb01538f980e3642bd Oct 31 19:19:14.226189 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:19:14.226172 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb7495b_ef69_455c_99aa_22389d8c6d6e.slice/crio-84fa1d57ccf00de8a4e885bb16678ee9716caaeaa5dfb50320f6d40c3ed5911c WatchSource:0}: Error finding container 84fa1d57ccf00de8a4e885bb16678ee9716caaeaa5dfb50320f6d40c3ed5911c: Status 404 returned error can't find the container with id 84fa1d57ccf00de8a4e885bb16678ee9716caaeaa5dfb50320f6d40c3ed5911c Oct 31 19:19:14.237331 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.237311 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9q7w4\" (UID: \"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:14.237420 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.237384 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-28mzh\" (UID: \"9cf7d7f5-3936-45b3-ba2c-597cf99cfb89\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-28mzh" Oct 31 19:19:14.237420 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.237405 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:14.237495 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.237430 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:14.237495 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.237465 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:14.237550 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.237495 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-d4kds\" (UID: \"4cd15118-0a59-4122-bdf1-56710c463262\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" Oct 31 19:19:14.237550 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.237541 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:14.237702 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:14.237685 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle podName:70c870bc-c9e6-4d7e-bc99-ba556ce31cde nodeName:}" failed. No retries permitted until 2026-10-31 19:19:15.237664908 +0000 UTC m=+32.051759140 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle") pod "router-default-8c75d8d7f-tzzdl" (UID: "70c870bc-c9e6-4d7e-bc99-ba556ce31cde") : configmap references non-existent config key: service-ca.crt Oct 31 19:19:14.237816 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:14.237713 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 19:19:14.237816 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:14.237762 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs podName:70c870bc-c9e6-4d7e-bc99-ba556ce31cde nodeName:}" failed. No retries permitted until 2026-10-31 19:19:15.237749972 +0000 UTC m=+32.051844182 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs") pod "router-default-8c75d8d7f-tzzdl" (UID: "70c870bc-c9e6-4d7e-bc99-ba556ce31cde") : secret "router-metrics-certs-default" not found Oct 31 19:19:14.237816 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:14.237763 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 19:19:14.237816 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:14.237776 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8ccd4bc6-xlml9: secret "image-registry-tls" not found Oct 31 19:19:14.237816 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:14.237796 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls podName:3a063ed7-8ce0-414e-8bbf-7733c464b507 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:15.237790186 +0000 UTC m=+32.051884394 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls") pod "image-registry-5c8ccd4bc6-xlml9" (UID: "3a063ed7-8ce0-414e-8bbf-7733c464b507") : secret "image-registry-tls" not found Oct 31 19:19:14.237816 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:14.237798 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 19:19:14.237816 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:14.237807 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f878896cc-hzn9v: secret "image-registry-tls" not found Oct 31 19:19:14.238194 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:14.237831 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls podName:eb1a9bea-399b-4281-9ada-a232790cc713 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:15.237823502 +0000 UTC m=+32.051917712 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls") pod "image-registry-6f878896cc-hzn9v" (UID: "eb1a9bea-399b-4281-9ada-a232790cc713") : secret "image-registry-tls" not found Oct 31 19:19:14.238194 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:14.237835 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 19:19:14.238194 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:14.237854 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls podName:f4d13f0a-4f53-4659-9a4e-0e1020f98b2e nodeName:}" failed. No retries permitted until 2026-10-31 19:19:15.237848764 +0000 UTC m=+32.051943017 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-9q7w4" (UID: "f4d13f0a-4f53-4659-9a4e-0e1020f98b2e") : secret "samples-operator-tls" not found Oct 31 19:19:14.238194 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:14.237867 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 19:19:14.238194 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:14.237887 2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 19:19:14.238194 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:14.237889 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls podName:4cd15118-0a59-4122-bdf1-56710c463262 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:15.237882261 +0000 UTC m=+32.051976470 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-d4kds" (UID: "4cd15118-0a59-4122-bdf1-56710c463262") : secret "cluster-monitoring-operator-tls" not found Oct 31 19:19:14.238194 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:14.237939 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert podName:9cf7d7f5-3936-45b3-ba2c-597cf99cfb89 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:15.237903017 +0000 UTC m=+32.051997275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-28mzh" (UID: "9cf7d7f5-3936-45b3-ba2c-597cf99cfb89") : secret "networking-console-plugin-cert" not found Oct 31 19:19:14.270103 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.270029 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-6vpwc"] Oct 31 19:19:14.274397 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.274371 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n"] Oct 31 19:19:14.275835 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:19:14.275807 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e07820_d888_4138_a41a_59ff4565accd.slice/crio-1c6926b25843cf9433e0d70f29db2aed8179291a04d03e2f5add0e961325899f WatchSource:0}: Error finding container 1c6926b25843cf9433e0d70f29db2aed8179291a04d03e2f5add0e961325899f: Status 404 returned error can't find the container with id 1c6926b25843cf9433e0d70f29db2aed8179291a04d03e2f5add0e961325899f Oct 31 19:19:14.277122 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:19:14.276984 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce261ba_08ff_418a_b3b4_b7e25e701bd8.slice/crio-6db78742cdd8d96d0c41b256e9d58eb82314da171eae09cb06fc9629394db69f WatchSource:0}: Error finding container 6db78742cdd8d96d0c41b256e9d58eb82314da171eae09cb06fc9629394db69f: Status 404 returned error can't find the container with id 6db78742cdd8d96d0c41b256e9d58eb82314da171eae09cb06fc9629394db69f Oct 31 19:19:14.286636 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.286601 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-l84pf"] Oct 31 19:19:14.289036 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.289016 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q8j58"] Oct 31 19:19:14.289727 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:19:14.289709 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e84edf8_6e35_489b_b19b_17a5179e666e.slice/crio-69226f4df7c0decbee73876af4ddd5ed0647b62ae34de6e1094c79717bbcda3b WatchSource:0}: Error finding container 69226f4df7c0decbee73876af4ddd5ed0647b62ae34de6e1094c79717bbcda3b: Status 404 returned error can't find the container with id 69226f4df7c0decbee73876af4ddd5ed0647b62ae34de6e1094c79717bbcda3b Oct 31 19:19:14.291902 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:19:14.291876 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92aaddbc_6268_443c_b3bd_9f74ab6b9a4b.slice/crio-58a3a53ca25510944bc534315dd2670e98583702f3bb6534584d79668465ad60 WatchSource:0}: Error finding container 58a3a53ca25510944bc534315dd2670e98583702f3bb6534584d79668465ad60: Status 404 returned error can't find the container with id 58a3a53ca25510944bc534315dd2670e98583702f3bb6534584d79668465ad60 Oct 31 19:19:14.339242 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.338881 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5418e7b8-1471-4aa5-ad70-5d60b87c572a-metrics-tls\") pod \"dns-default-5nm2g\" (UID: \"5418e7b8-1471-4aa5-ad70-5d60b87c572a\") " pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:14.339242 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:14.339141 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 19:19:14.339242 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:14.339208 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5418e7b8-1471-4aa5-ad70-5d60b87c572a-metrics-tls podName:5418e7b8-1471-4aa5-ad70-5d60b87c572a nodeName:}" failed. No retries permitted until 2026-10-31 19:19:15.339181211 +0000 UTC m=+32.153275434 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5418e7b8-1471-4aa5-ad70-5d60b87c572a-metrics-tls") pod "dns-default-5nm2g" (UID: "5418e7b8-1471-4aa5-ad70-5d60b87c572a") : secret "dns-default-metrics-tls" not found Oct 31 19:19:14.731765 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.731730 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:19:14.731765 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.731769 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:19:14.732548 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.731779 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:19:14.743802 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.743778 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nngmt\"" Oct 31 19:19:14.743802 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.743788 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jd7hq\"" Oct 31 19:19:14.743997 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.743880 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 19:19:14.743997 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.743886 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 19:19:14.743997 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.743949 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4hvl4\"" Oct 31 19:19:14.988040 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.987959 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n" event={"ID":"4ce261ba-08ff-418a-b3b4-b7e25e701bd8","Type":"ContainerStarted","Data":"6db78742cdd8d96d0c41b256e9d58eb82314da171eae09cb06fc9629394db69f"} Oct 31 19:19:14.990402 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.990372 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" event={"ID":"76e07820-d888-4138-a41a-59ff4565accd","Type":"ContainerStarted","Data":"1c6926b25843cf9433e0d70f29db2aed8179291a04d03e2f5add0e961325899f"} Oct 31 19:19:14.992073 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.992009 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgrs8" event={"ID":"ffb7495b-ef69-455c-99aa-22389d8c6d6e","Type":"ContainerStarted","Data":"84fa1d57ccf00de8a4e885bb16678ee9716caaeaa5dfb50320f6d40c3ed5911c"} Oct 31 19:19:14.993480 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.993450 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mtmp6" event={"ID":"599fc0e5-05d8-46b0-90b3-60818a371161","Type":"ContainerStarted","Data":"ada53789e04a83feca4c30151511cf86a4a6d7df24944c1c83b863ced5a2136c"} Oct 31 19:19:14.994866 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.994843 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm" event={"ID":"ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f","Type":"ContainerStarted","Data":"f3b2be88f679fe1cf5dd30b6b6873254fe93aa6dcff09dbb01538f980e3642bd"} Oct 31 19:19:14.997270 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.997199 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4mdb2" event={"ID":"3a7f7d6d-958c-4973-9289-97f45526a431","Type":"ContainerStarted","Data":"faab4fbfbd91c96325b42f8c8a06a858bd2b77b473ee44b7a0413f9fce01926a"} Oct 31 19:19:14.997270 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.997239 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4mdb2" event={"ID":"3a7f7d6d-958c-4973-9289-97f45526a431","Type":"ContainerStarted","Data":"0a7f85425d882bdc757d28a51639f891ce7c284d793f8bfb07690ada43662c58"} Oct 31 19:19:14.999977 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.999955 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xmfjb" event={"ID":"1cc2220b-f30e-4879-b2cb-b2dd8e53b9f0","Type":"ContainerStarted","Data":"8ea8feaa8aab45a0b019dd7d5a1923834eac6002b29726d3bdfa444d896cab27"} Oct 31 19:19:15.000068 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:14.999989 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xmfjb" event={"ID":"1cc2220b-f30e-4879-b2cb-b2dd8e53b9f0","Type":"ContainerStarted","Data":"336ffea7ff08409e19669798ffcc44ea0d354964196e08aafc954f9396d4ca05"} Oct 31 19:19:15.003519 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:15.003495 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-vszkf" event={"ID":"211efbea-29d0-4b2f-8c9e-6f3badcf77e5","Type":"ContainerStarted","Data":"78a98a1cf8579fd895716d7f907d4c1785947bbc727acb8cb09ba79987aec888"} Oct 31 19:19:15.003612 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:15.003536 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-vszkf" event={"ID":"211efbea-29d0-4b2f-8c9e-6f3badcf77e5","Type":"ContainerStarted","Data":"860bdd126bdb23ddbab4d48802a3be431f2d92a95f5d5716ba177928be8c2092"} Oct 31 19:19:15.006600 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:15.006565 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q8j58" event={"ID":"92aaddbc-6268-443c-b3bd-9f74ab6b9a4b","Type":"ContainerStarted","Data":"58a3a53ca25510944bc534315dd2670e98583702f3bb6534584d79668465ad60"} Oct 31 19:19:15.007840 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:15.007822 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l84pf" event={"ID":"3e84edf8-6e35-489b-b19b-17a5179e666e","Type":"ContainerStarted","Data":"69226f4df7c0decbee73876af4ddd5ed0647b62ae34de6e1094c79717bbcda3b"} Oct 31 19:19:15.018289 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:15.017240 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-4mdb2" podStartSLOduration=410.017230378 podStartE2EDuration="6m50.017230378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:15.015498442 +0000 UTC m=+31.829592683" watchObservedRunningTime="2026-10-31 19:19:15.017230378 +0000 UTC m=+31.831324610" Oct 31 19:19:15.060740 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:15.059588 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-vszkf" podStartSLOduration=410.059568142 podStartE2EDuration="6m50.059568142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:15.034410651 +0000 UTC m=+31.848504882" watchObservedRunningTime="2026-10-31 19:19:15.059568142 +0000 UTC m=+31.873662374" Oct 31 19:19:15.060740 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:15.060082 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-xmfjb" podStartSLOduration=410.060068998 podStartE2EDuration="6m50.060068998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:15.058731062 +0000 UTC m=+31.872825294" watchObservedRunningTime="2026-10-31 19:19:15.060068998 +0000 UTC m=+31.874163234" Oct 31 19:19:15.248765 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:15.248670 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-d4kds\" (UID: \"4cd15118-0a59-4122-bdf1-56710c463262\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" Oct 31 19:19:15.248765 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:15.248748 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:15.248980 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:15.248788 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9q7w4\" (UID: \"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:15.248980 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:15.248879 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-28mzh\" (UID: \"9cf7d7f5-3936-45b3-ba2c-597cf99cfb89\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-28mzh" Oct 31 19:19:15.248980 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:15.248923 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:15.248980 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:15.248955 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:15.249178 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:15.249014 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:15.249227 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:15.249180 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 19:19:15.249227 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:15.249196 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f878896cc-hzn9v: secret "image-registry-tls" not found Oct 31 19:19:15.249318 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:15.249255 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls podName:eb1a9bea-399b-4281-9ada-a232790cc713 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:17.249236593 +0000 UTC m=+34.063330826 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls") pod "image-registry-6f878896cc-hzn9v" (UID: "eb1a9bea-399b-4281-9ada-a232790cc713") : secret "image-registry-tls" not found Oct 31 19:19:15.249841 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:15.249494 2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 19:19:15.249841 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:15.249559 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert podName:9cf7d7f5-3936-45b3-ba2c-597cf99cfb89 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:17.249541255 +0000 UTC m=+34.063635478 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-28mzh" (UID: "9cf7d7f5-3936-45b3-ba2c-597cf99cfb89") : secret "networking-console-plugin-cert" not found Oct 31 19:19:15.249841 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:15.249619 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 19:19:15.249841 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:15.249654 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls podName:4cd15118-0a59-4122-bdf1-56710c463262 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:17.249643249 +0000 UTC m=+34.063737473 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-d4kds" (UID: "4cd15118-0a59-4122-bdf1-56710c463262") : secret "cluster-monitoring-operator-tls" not found Oct 31 19:19:15.249841 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:15.249673 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 19:19:15.249841 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:15.249717 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle podName:70c870bc-c9e6-4d7e-bc99-ba556ce31cde nodeName:}" failed. No retries permitted until 2026-10-31 19:19:17.249707158 +0000 UTC m=+34.063801382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle") pod "router-default-8c75d8d7f-tzzdl" (UID: "70c870bc-c9e6-4d7e-bc99-ba556ce31cde") : configmap references non-existent config key: service-ca.crt Oct 31 19:19:15.249841 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:15.249735 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs podName:70c870bc-c9e6-4d7e-bc99-ba556ce31cde nodeName:}" failed. No retries permitted until 2026-10-31 19:19:17.249725694 +0000 UTC m=+34.063819907 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs") pod "router-default-8c75d8d7f-tzzdl" (UID: "70c870bc-c9e6-4d7e-bc99-ba556ce31cde") : secret "router-metrics-certs-default" not found Oct 31 19:19:15.249841 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:15.249769 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 19:19:15.249841 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:15.249781 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8ccd4bc6-xlml9: secret "image-registry-tls" not found Oct 31 19:19:15.249841 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:15.249787 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 19:19:15.249841 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:15.249815 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls podName:3a063ed7-8ce0-414e-8bbf-7733c464b507 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:17.249801152 +0000 UTC m=+34.063895369 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls") pod "image-registry-5c8ccd4bc6-xlml9" (UID: "3a063ed7-8ce0-414e-8bbf-7733c464b507") : secret "image-registry-tls" not found Oct 31 19:19:15.249841 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:15.249833 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls podName:f4d13f0a-4f53-4659-9a4e-0e1020f98b2e nodeName:}" failed. No retries permitted until 2026-10-31 19:19:17.249823381 +0000 UTC m=+34.063917590 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-9q7w4" (UID: "f4d13f0a-4f53-4659-9a4e-0e1020f98b2e") : secret "samples-operator-tls" not found Oct 31 19:19:15.350499 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:15.349741 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5418e7b8-1471-4aa5-ad70-5d60b87c572a-metrics-tls\") pod \"dns-default-5nm2g\" (UID: \"5418e7b8-1471-4aa5-ad70-5d60b87c572a\") " pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:15.350499 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:15.350070 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 19:19:15.350499 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:15.350131 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5418e7b8-1471-4aa5-ad70-5d60b87c572a-metrics-tls podName:5418e7b8-1471-4aa5-ad70-5d60b87c572a nodeName:}" failed. No retries permitted until 2026-10-31 19:19:17.350114526 +0000 UTC m=+34.164208753 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5418e7b8-1471-4aa5-ad70-5d60b87c572a-metrics-tls") pod "dns-default-5nm2g" (UID: "5418e7b8-1471-4aa5-ad70-5d60b87c572a") : secret "dns-default-metrics-tls" not found Oct 31 19:19:17.272291 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:17.272186 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-28mzh\" (UID: \"9cf7d7f5-3936-45b3-ba2c-597cf99cfb89\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-28mzh" Oct 31 19:19:17.272291 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:17.272243 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:17.272291 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:17.272280 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:17.272836 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:17.272337 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:17.272836 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:17.272376 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-d4kds\" (UID: \"4cd15118-0a59-4122-bdf1-56710c463262\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" Oct 31 19:19:17.272836 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:17.272419 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:17.272836 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:17.272474 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9q7w4\" (UID: \"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:17.272836 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:17.272644 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 19:19:17.272836 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:17.272705 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls podName:f4d13f0a-4f53-4659-9a4e-0e1020f98b2e nodeName:}" failed. No retries permitted until 2026-10-31 19:19:21.272687659 +0000 UTC m=+38.086781887 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-9q7w4" (UID: "f4d13f0a-4f53-4659-9a4e-0e1020f98b2e") : secret "samples-operator-tls" not found Oct 31 19:19:17.273143 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:17.273018 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 19:19:17.273143 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:17.273032 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f878896cc-hzn9v: secret "image-registry-tls" not found Oct 31 19:19:17.273143 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:17.273120 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls podName:eb1a9bea-399b-4281-9ada-a232790cc713 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:21.273061393 +0000 UTC m=+38.087155617 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls") pod "image-registry-6f878896cc-hzn9v" (UID: "eb1a9bea-399b-4281-9ada-a232790cc713") : secret "image-registry-tls" not found Oct 31 19:19:17.273265 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:17.273181 2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 19:19:17.273265 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:17.273218 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert podName:9cf7d7f5-3936-45b3-ba2c-597cf99cfb89 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:21.273206823 +0000 UTC m=+38.087301046 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-28mzh" (UID: "9cf7d7f5-3936-45b3-ba2c-597cf99cfb89") : secret "networking-console-plugin-cert" not found Oct 31 19:19:17.273359 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:17.273267 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 19:19:17.273359 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:17.273298 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs podName:70c870bc-c9e6-4d7e-bc99-ba556ce31cde nodeName:}" failed. No retries permitted until 2026-10-31 19:19:21.273288087 +0000 UTC m=+38.087382313 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs") pod "router-default-8c75d8d7f-tzzdl" (UID: "70c870bc-c9e6-4d7e-bc99-ba556ce31cde") : secret "router-metrics-certs-default" not found Oct 31 19:19:17.273461 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:17.273393 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 19:19:17.273461 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:17.273403 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8ccd4bc6-xlml9: secret "image-registry-tls" not found Oct 31 19:19:17.273461 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:17.273442 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls podName:3a063ed7-8ce0-414e-8bbf-7733c464b507 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:21.273429612 +0000 UTC m=+38.087523835 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls") pod "image-registry-5c8ccd4bc6-xlml9" (UID: "3a063ed7-8ce0-414e-8bbf-7733c464b507") : secret "image-registry-tls" not found Oct 31 19:19:17.273591 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:17.273509 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle podName:70c870bc-c9e6-4d7e-bc99-ba556ce31cde nodeName:}" failed. No retries permitted until 2026-10-31 19:19:21.273499651 +0000 UTC m=+38.087593874 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle") pod "router-default-8c75d8d7f-tzzdl" (UID: "70c870bc-c9e6-4d7e-bc99-ba556ce31cde") : configmap references non-existent config key: service-ca.crt Oct 31 19:19:17.273591 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:17.273564 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 19:19:17.273688 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:17.273594 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls podName:4cd15118-0a59-4122-bdf1-56710c463262 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:21.273584111 +0000 UTC m=+38.087678339 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-d4kds" (UID: "4cd15118-0a59-4122-bdf1-56710c463262") : secret "cluster-monitoring-operator-tls" not found Oct 31 19:19:17.373199 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:17.373161 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5418e7b8-1471-4aa5-ad70-5d60b87c572a-metrics-tls\") pod \"dns-default-5nm2g\" (UID: \"5418e7b8-1471-4aa5-ad70-5d60b87c572a\") " pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:17.373869 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:17.373340 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 19:19:17.373869 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:17.373420 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5418e7b8-1471-4aa5-ad70-5d60b87c572a-metrics-tls podName:5418e7b8-1471-4aa5-ad70-5d60b87c572a nodeName:}" failed. No retries permitted until 2026-10-31 19:19:21.373401208 +0000 UTC m=+38.187495432 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5418e7b8-1471-4aa5-ad70-5d60b87c572a-metrics-tls") pod "dns-default-5nm2g" (UID: "5418e7b8-1471-4aa5-ad70-5d60b87c572a") : secret "dns-default-metrics-tls" not found Oct 31 19:19:17.474617 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:17.474334 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88a1fa4b-1191-434a-8e7c-783163ece24d-metrics-certs\") pod \"network-metrics-daemon-dfbrp\" (UID: \"88a1fa4b-1191-434a-8e7c-783163ece24d\") " pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:19:17.474617 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:17.474432 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9255a1d5-2514-441e-b942-6c39843e38f8-original-pull-secret\") pod \"global-pull-secret-syncer-lhtkb\" (UID: \"9255a1d5-2514-441e-b942-6c39843e38f8\") " pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:19:17.475311 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:17.475288 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 19:19:17.475414 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:17.475368 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88a1fa4b-1191-434a-8e7c-783163ece24d-metrics-certs podName:88a1fa4b-1191-434a-8e7c-783163ece24d nodeName:}" failed. No retries permitted until 2026-10-31 19:19:49.47534858 +0000 UTC m=+66.289442802 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88a1fa4b-1191-434a-8e7c-783163ece24d-metrics-certs") pod "network-metrics-daemon-dfbrp" (UID: "88a1fa4b-1191-434a-8e7c-783163ece24d") : secret "metrics-daemon-secret" not found Oct 31 19:19:17.486305 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:17.486253 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9255a1d5-2514-441e-b942-6c39843e38f8-original-pull-secret\") pod \"global-pull-secret-syncer-lhtkb\" (UID: \"9255a1d5-2514-441e-b942-6c39843e38f8\") " pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:19:17.524307 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:17.523997 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:19:17.576216 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:17.575651 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7djgh\" (UniqueName: \"kubernetes.io/projected/23a10416-51b1-415f-b2fc-892b4d662b1f-kube-api-access-7djgh\") pod \"network-check-target-5mkxz\" (UID: \"23a10416-51b1-415f-b2fc-892b4d662b1f\") " pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:19:17.585350 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:17.585295 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7djgh\" (UniqueName: \"kubernetes.io/projected/23a10416-51b1-415f-b2fc-892b4d662b1f-kube-api-access-7djgh\") pod \"network-check-target-5mkxz\" (UID: \"23a10416-51b1-415f-b2fc-892b4d662b1f\") " pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:19:17.783274 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:17.783191 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:19:17.855630 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:17.855558 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lhtkb"] Oct 31 19:19:17.877408 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:19:17.877376 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9255a1d5_2514_441e_b942_6c39843e38f8.slice/crio-19544c188d1e4a6b98188a935a654c1b0e9d94c776c707dc733952c54f0e2067 WatchSource:0}: Error finding container 19544c188d1e4a6b98188a935a654c1b0e9d94c776c707dc733952c54f0e2067: Status 404 returned error can't find the container with id 19544c188d1e4a6b98188a935a654c1b0e9d94c776c707dc733952c54f0e2067 Oct 31 19:19:18.029206 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:18.029155 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lhtkb" event={"ID":"9255a1d5-2514-441e-b942-6c39843e38f8","Type":"ContainerStarted","Data":"19544c188d1e4a6b98188a935a654c1b0e9d94c776c707dc733952c54f0e2067"} Oct 31 19:19:18.031449 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:18.031412 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgrs8" event={"ID":"ffb7495b-ef69-455c-99aa-22389d8c6d6e","Type":"ContainerStarted","Data":"395ede5a6a8d1942ce5066eba4905e48b189843c5fd44f167f1b9f1fee5b740e"} Oct 31 19:19:18.051479 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:18.051356 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgrs8" podStartSLOduration=394.334779799 podStartE2EDuration="6m37.051337191s" totalImagesPullingTime="2.716557392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:18.050531668 +0000 UTC m=+34.864625899" watchObservedRunningTime="2026-10-31 19:19:18.051337191 +0000 UTC m=+34.865431424" Oct 31 19:19:18.112534 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:18.112491 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5mkxz"] Oct 31 19:19:18.117437 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:19:18.117389 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a10416_51b1_415f_b2fc_892b4d662b1f.slice/crio-d47104b37f449b2e3b4f416da67e4be44ab266e8aff8244d4804ec228a4fce84 WatchSource:0}: Error finding container d47104b37f449b2e3b4f416da67e4be44ab266e8aff8244d4804ec228a4fce84: Status 404 returned error can't find the container with id d47104b37f449b2e3b4f416da67e4be44ab266e8aff8244d4804ec228a4fce84 Oct 31 19:19:19.037316 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:19.037283 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5mkxz" event={"ID":"23a10416-51b1-415f-b2fc-892b4d662b1f","Type":"ContainerStarted","Data":"d47104b37f449b2e3b4f416da67e4be44ab266e8aff8244d4804ec228a4fce84"} Oct 31 19:19:20.042542 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.042504 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n" event={"ID":"4ce261ba-08ff-418a-b3b4-b7e25e701bd8","Type":"ContainerStarted","Data":"d2d75ee097b7130386bb3b641ea0ee1129eb5cf4404817e71fec2f1596fc2c45"} Oct 31 19:19:20.044585 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.044561 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/0.log" Oct 31 19:19:20.044718 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.044599 2566 generic.go:356] "Generic (PLEG): container finished" podID="76e07820-d888-4138-a41a-59ff4565accd" containerID="eb5ba63f298e0517756736894e7326ae52ee2b859ddc0b29ee9b50b14c45f51a" exitCode=255 Oct 31 19:19:20.044718 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.044665 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" event={"ID":"76e07820-d888-4138-a41a-59ff4565accd","Type":"ContainerDied","Data":"eb5ba63f298e0517756736894e7326ae52ee2b859ddc0b29ee9b50b14c45f51a"} Oct 31 19:19:20.044932 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.044901 2566 scope.go:122] "RemoveContainer" containerID="eb5ba63f298e0517756736894e7326ae52ee2b859ddc0b29ee9b50b14c45f51a" Oct 31 19:19:20.046882 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.046857 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mtmp6" event={"ID":"599fc0e5-05d8-46b0-90b3-60818a371161","Type":"ContainerStarted","Data":"fde1d7db146568648f3672e51dd2f91b660f40070603b600b78a777d07445d44"} Oct 31 19:19:20.049266 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.049216 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm" event={"ID":"ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f","Type":"ContainerStarted","Data":"cc31eee786c6c1ce374504d7227aa2d41018f11549d58a3d0a78202dbb3da80e"} Oct 31 19:19:20.085254 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.085208 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n" podStartSLOduration=392.859886653 podStartE2EDuration="6m38.085190161s" totalImagesPullingTime="5.225303508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:12:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:20.063100387 +0000 UTC m=+36.877194619" watchObservedRunningTime="2026-10-31 19:19:20.085190161 +0000 UTC m=+36.899284394" Oct 31 19:19:20.086151 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.086047 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm" podStartSLOduration=392.832989735 podStartE2EDuration="6m38.086037801s" totalImagesPullingTime="5.253048066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:12:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:20.085650246 +0000 UTC m=+36.899744481" watchObservedRunningTime="2026-10-31 19:19:20.086037801 +0000 UTC m=+36.900132014" Oct 31 19:19:20.442544 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.440601 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-mtmp6" podStartSLOduration=388.170570593 podStartE2EDuration="6m33.440583294s" totalImagesPullingTime="5.270012701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:20.14729267 +0000 UTC m=+36.961386903" watchObservedRunningTime="2026-10-31 19:19:20.440583294 +0000 UTC m=+37.254677526" Oct 31 19:19:20.442544 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.441290 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4fsp7"] Oct 31 19:19:20.460952 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.460905 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4fsp7" Oct 31 19:19:20.475715 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.475655 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mpqbj\"" Oct 31 19:19:20.477866 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.477811 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4fsp7"] Oct 31 19:19:20.507045 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.507017 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 19:19:20.507045 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.507018 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 19:19:20.541018 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.540988 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5nkwp"] Oct 31 19:19:20.557182 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.557155 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:20.559164 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.559137 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5nkwp"] Oct 31 19:19:20.575559 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.575511 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 19:19:20.575892 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.575855 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mxmqk\"" Oct 31 19:19:20.607319 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.607292 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 19:19:20.615350 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.615029 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7fw4\" (UniqueName: \"kubernetes.io/projected/7bce545d-6640-49e8-848c-aa9f8db19902-kube-api-access-l7fw4\") pod \"migrator-66df5d496b-4fsp7\" (UID: \"7bce545d-6640-49e8-848c-aa9f8db19902\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4fsp7" Oct 31 19:19:20.715718 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.715629 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn2tq\" (UniqueName: \"kubernetes.io/projected/4e729724-d310-4884-a2d0-54af5308124c-kube-api-access-fn2tq\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:20.715718 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.715688 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e729724-d310-4884-a2d0-54af5308124c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:20.715996 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.715768 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e729724-d310-4884-a2d0-54af5308124c-data-volume\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:20.715996 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.715841 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7fw4\" (UniqueName: \"kubernetes.io/projected/7bce545d-6640-49e8-848c-aa9f8db19902-kube-api-access-l7fw4\") pod \"migrator-66df5d496b-4fsp7\" (UID: \"7bce545d-6640-49e8-848c-aa9f8db19902\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4fsp7" Oct 31 19:19:20.715996 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.715862 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e729724-d310-4884-a2d0-54af5308124c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:20.715996 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.715902 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e729724-d310-4884-a2d0-54af5308124c-crio-socket\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:20.753607 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.753578 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7fw4\" (UniqueName: \"kubernetes.io/projected/7bce545d-6640-49e8-848c-aa9f8db19902-kube-api-access-l7fw4\") pod \"migrator-66df5d496b-4fsp7\" (UID: \"7bce545d-6640-49e8-848c-aa9f8db19902\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4fsp7" Oct 31 19:19:20.801954 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.801903 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4fsp7" Oct 31 19:19:20.816612 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.816581 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e729724-d310-4884-a2d0-54af5308124c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:20.816775 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.816631 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e729724-d310-4884-a2d0-54af5308124c-crio-socket\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:20.816775 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.816672 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn2tq\" (UniqueName: \"kubernetes.io/projected/4e729724-d310-4884-a2d0-54af5308124c-kube-api-access-fn2tq\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:20.816775 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.816694 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e729724-d310-4884-a2d0-54af5308124c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:20.816775 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:20.816729 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 19:19:20.816775 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.816749 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e729724-d310-4884-a2d0-54af5308124c-data-volume\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:20.817042 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.816779 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e729724-d310-4884-a2d0-54af5308124c-crio-socket\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:20.817042 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:20.816810 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e729724-d310-4884-a2d0-54af5308124c-insights-runtime-extractor-tls podName:4e729724-d310-4884-a2d0-54af5308124c nodeName:}" failed. No retries permitted until 2026-10-31 19:19:21.316786404 +0000 UTC m=+38.130880632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4e729724-d310-4884-a2d0-54af5308124c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5nkwp" (UID: "4e729724-d310-4884-a2d0-54af5308124c") : secret "insights-runtime-extractor-tls" not found Oct 31 19:19:20.817042 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:20.816840 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 19:19:20.817042 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:20.816938 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4e729724-d310-4884-a2d0-54af5308124c-kube-rbac-proxy-cm podName:4e729724-d310-4884-a2d0-54af5308124c nodeName:}" failed. No retries permitted until 2026-10-31 19:19:21.316897625 +0000 UTC m=+38.130991841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4e729724-d310-4884-a2d0-54af5308124c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5nkwp" (UID: "4e729724-d310-4884-a2d0-54af5308124c") : configmap "kube-rbac-proxy" not found Oct 31 19:19:20.817250 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.817133 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e729724-d310-4884-a2d0-54af5308124c-data-volume\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:20.850160 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:20.850133 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn2tq\" (UniqueName: \"kubernetes.io/projected/4e729724-d310-4884-a2d0-54af5308124c-kube-api-access-fn2tq\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:21.055186 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:21.055068 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q8j58" event={"ID":"92aaddbc-6268-443c-b3bd-9f74ab6b9a4b","Type":"ContainerStarted","Data":"9739701b61a37859d67b956104e6b29b121d140762343f151aa159160f3718a3"} Oct 31 19:19:21.056835 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:21.056807 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l84pf" event={"ID":"3e84edf8-6e35-489b-b19b-17a5179e666e","Type":"ContainerStarted","Data":"a688c16c8575642c1e13ebeebc18b0f16a41208d3654af75b746bbd0f0bb6099"} Oct 31 19:19:21.058719 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:21.058692 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/1.log" Oct 31 19:19:21.059266 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:21.059241 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/0.log" Oct 31 19:19:21.059370 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:21.059282 2566 generic.go:356] "Generic (PLEG): container finished" podID="76e07820-d888-4138-a41a-59ff4565accd" containerID="3a9fb18642a3e4ea0da071d9b3009cfddbd0e22c650b9a07b4a1d921bd019368" exitCode=255 Oct 31 19:19:21.059424 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:21.059377 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" event={"ID":"76e07820-d888-4138-a41a-59ff4565accd","Type":"ContainerDied","Data":"3a9fb18642a3e4ea0da071d9b3009cfddbd0e22c650b9a07b4a1d921bd019368"} Oct 31 19:19:21.059424 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:21.059418 2566 scope.go:122] "RemoveContainer" containerID="eb5ba63f298e0517756736894e7326ae52ee2b859ddc0b29ee9b50b14c45f51a" Oct 31 19:19:21.060071 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:21.059798 2566 scope.go:122] "RemoveContainer" containerID="3a9fb18642a3e4ea0da071d9b3009cfddbd0e22c650b9a07b4a1d921bd019368" Oct 31 19:19:21.060071 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:21.060020 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6vpwc_openshift-console-operator(76e07820-d888-4138-a41a-59ff4565accd)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" podUID="76e07820-d888-4138-a41a-59ff4565accd" Oct 31 19:19:21.084680 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:21.084634 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-q8j58" podStartSLOduration=2.3106088 podStartE2EDuration="8.084622004s" totalImagesPullingTime="5.774013204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:21.083439607 +0000 UTC m=+37.897533839" watchObservedRunningTime="2026-10-31 19:19:21.084622004 +0000 UTC m=+37.898716240" Oct 31 19:19:21.119837 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:21.119773 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4fsp7"] Oct 31 19:19:21.122061 ip-10-0-142-116 kubenswrapper[2566]: W1031 19:19:21.122029 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bce545d_6640_49e8_848c_aa9f8db19902.slice/crio-c54a9a71ab9d46f891569cbf6da57803dd50a5527f313b59c4b1353d218e696a WatchSource:0}: Error finding container c54a9a71ab9d46f891569cbf6da57803dd50a5527f313b59c4b1353d218e696a: Status 404 returned error can't find the container with id c54a9a71ab9d46f891569cbf6da57803dd50a5527f313b59c4b1353d218e696a Oct 31 19:19:21.322727 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:21.322645 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:21.322727 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:21.322692 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9q7w4\" (UID: \"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:21.322899 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:21.322784 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 19:19:21.322899 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:21.322842 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls podName:f4d13f0a-4f53-4659-9a4e-0e1020f98b2e nodeName:}" failed. No retries permitted until 2026-10-31 19:19:29.322826928 +0000 UTC m=+46.136921154 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-9q7w4" (UID: "f4d13f0a-4f53-4659-9a4e-0e1020f98b2e") : secret "samples-operator-tls" not found Oct 31 19:19:21.322899 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:21.322857 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e729724-d310-4884-a2d0-54af5308124c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:21.322899 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:21.322883 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-28mzh\" (UID: \"9cf7d7f5-3936-45b3-ba2c-597cf99cfb89\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-28mzh" Oct 31 19:19:21.323101 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:21.322924 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle podName:70c870bc-c9e6-4d7e-bc99-ba556ce31cde nodeName:}" failed. No retries permitted until 2026-10-31 19:19:29.322896842 +0000 UTC m=+46.136991056 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle") pod "router-default-8c75d8d7f-tzzdl" (UID: "70c870bc-c9e6-4d7e-bc99-ba556ce31cde") : configmap references non-existent config key: service-ca.crt Oct 31 19:19:21.323101 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:21.322961 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:21.323101 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:21.322968 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 19:19:21.323101 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:21.322976 2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 19:19:21.323101 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:21.322987 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:21.323101 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:21.323006 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e729724-d310-4884-a2d0-54af5308124c-insights-runtime-extractor-tls podName:4e729724-d310-4884-a2d0-54af5308124c nodeName:}" failed. No retries permitted until 2026-10-31 19:19:22.322993312 +0000 UTC m=+39.137087540 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4e729724-d310-4884-a2d0-54af5308124c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5nkwp" (UID: "4e729724-d310-4884-a2d0-54af5308124c") : secret "insights-runtime-extractor-tls" not found Oct 31 19:19:21.323101 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:21.323030 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert podName:9cf7d7f5-3936-45b3-ba2c-597cf99cfb89 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:29.323017469 +0000 UTC m=+46.137111692 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-28mzh" (UID: "9cf7d7f5-3936-45b3-ba2c-597cf99cfb89") : secret "networking-console-plugin-cert" not found Oct 31 19:19:21.323101 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:21.323050 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 19:19:21.323101 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:21.323060 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8ccd4bc6-xlml9: secret "image-registry-tls" not found Oct 31 19:19:21.323101 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:21.323083 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e729724-d310-4884-a2d0-54af5308124c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:21.323101 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:21.323088 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls podName:3a063ed7-8ce0-414e-8bbf-7733c464b507 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:29.323080036 +0000 UTC m=+46.137174267 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls") pod "image-registry-5c8ccd4bc6-xlml9" (UID: "3a063ed7-8ce0-414e-8bbf-7733c464b507") : secret "image-registry-tls" not found Oct 31 19:19:21.323101 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:21.323084 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 19:19:21.323101 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:21.323109 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:21.323510 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:21.323115 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs podName:70c870bc-c9e6-4d7e-bc99-ba556ce31cde nodeName:}" failed. No retries permitted until 2026-10-31 19:19:29.323108773 +0000 UTC m=+46.137203027 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs") pod "router-default-8c75d8d7f-tzzdl" (UID: "70c870bc-c9e6-4d7e-bc99-ba556ce31cde") : secret "router-metrics-certs-default" not found Oct 31 19:19:21.323510 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:21.323128 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 19:19:21.323510 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:21.323144 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-d4kds\" (UID: \"4cd15118-0a59-4122-bdf1-56710c463262\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" Oct 31 19:19:21.323510 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:21.323151 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 19:19:21.323510 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:21.323161 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f878896cc-hzn9v: secret "image-registry-tls" not found Oct 31 19:19:21.323510 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:21.323182 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4e729724-d310-4884-a2d0-54af5308124c-kube-rbac-proxy-cm podName:4e729724-d310-4884-a2d0-54af5308124c nodeName:}" failed. No retries permitted until 2026-10-31 19:19:22.323165376 +0000 UTC m=+39.137259601 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4e729724-d310-4884-a2d0-54af5308124c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5nkwp" (UID: "4e729724-d310-4884-a2d0-54af5308124c") : configmap "kube-rbac-proxy" not found Oct 31 19:19:21.323510 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:21.323203 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls podName:eb1a9bea-399b-4281-9ada-a232790cc713 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:29.323192479 +0000 UTC m=+46.137286722 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls") pod "image-registry-6f878896cc-hzn9v" (UID: "eb1a9bea-399b-4281-9ada-a232790cc713") : secret "image-registry-tls" not found Oct 31 19:19:21.323510 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:21.323219 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 19:19:21.323510 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:21.323241 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls podName:4cd15118-0a59-4122-bdf1-56710c463262 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:29.323234543 +0000 UTC m=+46.137328753 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-d4kds" (UID: "4cd15118-0a59-4122-bdf1-56710c463262") : secret "cluster-monitoring-operator-tls" not found Oct 31 19:19:21.424216 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:21.424187 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5418e7b8-1471-4aa5-ad70-5d60b87c572a-metrics-tls\") pod \"dns-default-5nm2g\" (UID: \"5418e7b8-1471-4aa5-ad70-5d60b87c572a\") " pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:21.424357 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:21.424327 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 19:19:21.424405 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:21.424384 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5418e7b8-1471-4aa5-ad70-5d60b87c572a-metrics-tls podName:5418e7b8-1471-4aa5-ad70-5d60b87c572a nodeName:}" failed. No retries permitted until 2026-10-31 19:19:29.424369594 +0000 UTC m=+46.238463803 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5418e7b8-1471-4aa5-ad70-5d60b87c572a-metrics-tls") pod "dns-default-5nm2g" (UID: "5418e7b8-1471-4aa5-ad70-5d60b87c572a") : secret "dns-default-metrics-tls" not found Oct 31 19:19:22.066659 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:22.066581 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4fsp7" event={"ID":"7bce545d-6640-49e8-848c-aa9f8db19902","Type":"ContainerStarted","Data":"c54a9a71ab9d46f891569cbf6da57803dd50a5527f313b59c4b1353d218e696a"} Oct 31 19:19:22.070253 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:22.070232 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/1.log" Oct 31 19:19:22.070584 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:22.070564 2566 scope.go:122] "RemoveContainer" containerID="3a9fb18642a3e4ea0da071d9b3009cfddbd0e22c650b9a07b4a1d921bd019368" Oct 31 19:19:22.070817 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:22.070791 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6vpwc_openshift-console-operator(76e07820-d888-4138-a41a-59ff4565accd)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" podUID="76e07820-d888-4138-a41a-59ff4565accd" Oct 31 19:19:22.072322 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:22.071882 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5mkxz" event={"ID":"23a10416-51b1-415f-b2fc-892b4d662b1f","Type":"ContainerStarted","Data":"1c25140b3e48674306a8741127cd8a5bf1b98a360c10dee02621d4fdbb0797d4"} Oct 31 19:19:22.072322 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:22.072010 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:19:22.096483 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:22.096302 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l84pf" podStartSLOduration=361.288987777 podStartE2EDuration="6m7.096282196s" totalImagesPullingTime="5.807294419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:13:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:21.154031092 +0000 UTC m=+37.968125323" watchObservedRunningTime="2026-10-31 19:19:22.096282196 +0000 UTC m=+38.910376519" Oct 31 19:19:22.113965 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:22.113899 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5mkxz" podStartSLOduration=34.610603978 podStartE2EDuration="38.113881556s" totalImagesPullingTime="3.503277578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:22.11365378 +0000 UTC m=+38.927748012" watchObservedRunningTime="2026-10-31 19:19:22.113881556 +0000 UTC m=+38.927975788" Oct 31 19:19:22.334669 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:22.334581 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e729724-d310-4884-a2d0-54af5308124c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:22.334853 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:22.334714 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e729724-d310-4884-a2d0-54af5308124c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:22.334853 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:22.334755 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 19:19:22.334853 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:22.334820 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e729724-d310-4884-a2d0-54af5308124c-insights-runtime-extractor-tls podName:4e729724-d310-4884-a2d0-54af5308124c nodeName:}" failed. No retries permitted until 2026-10-31 19:19:24.334804743 +0000 UTC m=+41.148898968 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4e729724-d310-4884-a2d0-54af5308124c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5nkwp" (UID: "4e729724-d310-4884-a2d0-54af5308124c") : secret "insights-runtime-extractor-tls" not found Oct 31 19:19:22.335055 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:22.334863 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 19:19:22.335055 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:22.334928 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4e729724-d310-4884-a2d0-54af5308124c-kube-rbac-proxy-cm podName:4e729724-d310-4884-a2d0-54af5308124c nodeName:}" failed. No retries permitted until 2026-10-31 19:19:24.334902273 +0000 UTC m=+41.148996497 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4e729724-d310-4884-a2d0-54af5308124c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5nkwp" (UID: "4e729724-d310-4884-a2d0-54af5308124c") : configmap "kube-rbac-proxy" not found Oct 31 19:19:23.077583 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:23.077504 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4fsp7" event={"ID":"7bce545d-6640-49e8-848c-aa9f8db19902","Type":"ContainerStarted","Data":"3379616a1fcef23013c28a41575f0472c36e42d4674065cd7d02fabc920b6530"} Oct 31 19:19:23.096758 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:23.096705 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4fsp7" podStartSLOduration=1.492340523 podStartE2EDuration="3.096690702s" totalImagesPullingTime="1.604350179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:23.095569758 +0000 UTC m=+39.909663990" watchObservedRunningTime="2026-10-31 19:19:23.096690702 +0000 UTC m=+39.910784932" Oct 31 19:19:23.922996 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:23.922959 2566 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:23.923198 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:23.923034 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:23.923473 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:23.923452 2566 scope.go:122] "RemoveContainer" containerID="3a9fb18642a3e4ea0da071d9b3009cfddbd0e22c650b9a07b4a1d921bd019368" Oct 31 19:19:23.923700 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:23.923673 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6vpwc_openshift-console-operator(76e07820-d888-4138-a41a-59ff4565accd)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" podUID="76e07820-d888-4138-a41a-59ff4565accd" Oct 31 19:19:24.354677 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:24.354594 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e729724-d310-4884-a2d0-54af5308124c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:24.355117 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:24.354708 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e729724-d310-4884-a2d0-54af5308124c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:24.355117 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:24.354758 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 19:19:24.355117 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:24.354836 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e729724-d310-4884-a2d0-54af5308124c-insights-runtime-extractor-tls podName:4e729724-d310-4884-a2d0-54af5308124c nodeName:}" failed. No retries permitted until 2026-10-31 19:19:28.354815387 +0000 UTC m=+45.168909601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4e729724-d310-4884-a2d0-54af5308124c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5nkwp" (UID: "4e729724-d310-4884-a2d0-54af5308124c") : secret "insights-runtime-extractor-tls" not found Oct 31 19:19:24.355117 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:24.354842 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 19:19:24.355117 ip-10-0-142-116 kubenswrapper[2566]: E1031 19:19:24.354926 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4e729724-d310-4884-a2d0-54af5308124c-kube-rbac-proxy-cm podName:4e729724-d310-4884-a2d0-54af5308124c nodeName:}" failed. No retries permitted until 2026-10-31 19:19:28.354889553 +0000 UTC m=+45.168983776 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4e729724-d310-4884-a2d0-54af5308124c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5nkwp" (UID: "4e729724-d310-4884-a2d0-54af5308124c") : configmap "kube-rbac-proxy" not found Oct 31 19:19:25.518133 ip-10-0-142-116 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 19:19:25.518386 ip-10-0-142-116 kubenswrapper[2566]: I1031 19:19:25.517992 2566 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 19:19:25.537396 ip-10-0-142-116 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 19:19:25.537619 ip-10-0-142-116 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 19:19:25.537870 ip-10-0-142-116 systemd[1]: kubelet.service: Consumed 3.030s CPU time. Oct 31 19:19:25.553446 ip-10-0-142-116 systemd[1]: Starting Kubernetes Kubelet... Oct 31 19:19:25.667748 ip-10-0-142-116 kubenswrapper[4910]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 19:19:25.667748 ip-10-0-142-116 kubenswrapper[4910]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 19:19:25.667748 ip-10-0-142-116 kubenswrapper[4910]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 19:19:25.667748 ip-10-0-142-116 kubenswrapper[4910]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 19:19:25.669878 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669791 4910 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:19:25.669878 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669877 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:19:25.669878 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669881 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:19:25.669995 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669885 4910 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:19:25.669995 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669890 4910 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:19:25.669995 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669894 4910 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:19:25.669995 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669897 4910 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:19:25.669995 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669904 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:19:25.669995 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669919 4910 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:19:25.669995 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669923 4910 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:19:25.669995 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669925 4910 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:19:25.669995 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669928 4910 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:19:25.669995 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669931 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:19:25.669995 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669934 4910 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:19:25.669995 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669937 4910 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:19:25.669995 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669940 4910 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:19:25.669995 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669942 4910 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:19:25.669995 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669945 4910 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:19:25.669995 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669949 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:19:25.669995 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669952 4910 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:19:25.669995 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669955 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:19:25.669995 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669957 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:19:25.670462 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669960 4910 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:19:25.670462 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669963 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:19:25.670462 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669965 4910 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:19:25.670462 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669968 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:19:25.670462 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669971 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:19:25.670462 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669973 4910 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:19:25.670462 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669976 4910 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:19:25.670462 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669978 4910 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:19:25.670462 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669981 4910 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:19:25.670462 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669983 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:19:25.670462 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669986 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:19:25.670462 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669988 4910 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:19:25.670462 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669991 4910 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:19:25.670462 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669993 4910 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:19:25.670462 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669996 4910 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:19:25.670462 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.669999 4910 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:19:25.670462 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670004 4910 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:19:25.670462 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670009 4910 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:19:25.670462 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670013 4910 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:19:25.670943 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670016 4910 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:19:25.670943 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670018 4910 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:19:25.670943 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670021 4910 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:19:25.670943 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670024 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:19:25.670943 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670031 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:19:25.670943 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670034 4910 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:19:25.670943 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670037 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:19:25.670943 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670039 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:19:25.670943 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670042 4910 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:19:25.670943 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670045 4910 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:19:25.670943 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670047 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:19:25.670943 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670050 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:19:25.670943 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670052 4910 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:19:25.670943 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670055 4910 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:19:25.670943 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670057 4910 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:19:25.670943 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670060 4910 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:19:25.670943 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670063 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:19:25.670943 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670065 4910 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:19:25.670943 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670068 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:19:25.671413 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670070 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:19:25.671413 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670073 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:19:25.671413 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670088 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:19:25.671413 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670091 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:19:25.671413 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670094 4910 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:19:25.671413 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670097 4910 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:19:25.671413 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670100 4910 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:19:25.671413 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670103 4910 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:19:25.671413 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670105 4910 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:19:25.671413 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670108 4910 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:19:25.671413 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670111 4910 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:19:25.671413 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670113 4910 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:19:25.671413 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670117 4910 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:19:25.671413 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670119 4910 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:19:25.671413 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670122 4910 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:19:25.671413 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670125 4910 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:19:25.671413 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670127 4910 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:19:25.671413 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670132 4910 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:19:25.671413 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670135 4910 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:19:25.671413 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670137 4910 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:19:25.671902 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670140 4910 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:19:25.671902 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670142 4910 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:19:25.671902 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670145 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:19:25.671902 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670148 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:19:25.671902 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670150 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:19:25.671902 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670153 4910 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:19:25.671902 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670155 4910 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:19:25.671902 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670158 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:19:25.671902 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670160 4910 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:19:25.671902 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670163 4910 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:19:25.671902 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670167 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:19:25.671902 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670170 4910 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:19:25.671902 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670172 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:19:25.671902 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670175 4910 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:19:25.671902 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670177 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:19:25.671902 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670180 4910 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:19:25.671902 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670182 4910 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:19:25.671902 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670185 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:19:25.671902 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670188 4910 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:19:25.671902 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670191 4910 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:19:25.672422 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670193 4910 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:19:25.672422 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670196 4910 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:19:25.672422 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670199 4910 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:19:25.672422 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670201 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:19:25.672422 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670204 4910 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:19:25.672422 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670207 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:19:25.672422 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670992 4910 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:19:25.672422 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.670999 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:19:25.672422 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671002 4910 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:19:25.672422 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671007 4910 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:19:25.672422 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671010 4910 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:19:25.672422 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671013 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:19:25.672422 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671015 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:19:25.672422 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671018 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:19:25.672422 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671021 4910 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:19:25.672422 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671023 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:19:25.672422 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671026 4910 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:19:25.672422 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671029 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:19:25.672422 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671031 4910 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:19:25.672422 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671034 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:19:25.672973 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671036 4910 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:19:25.672973 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671039 4910 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:19:25.672973 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671041 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:19:25.672973 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671044 4910 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:19:25.672973 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671046 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:19:25.672973 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671050 4910 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:19:25.672973 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671052 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:19:25.672973 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671056 4910 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:19:25.672973 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671058 4910 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:19:25.672973 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671061 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:19:25.672973 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671063 4910 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:19:25.672973 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671066 4910 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:19:25.672973 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671069 4910 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:19:25.672973 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671071 4910 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:19:25.672973 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671074 4910 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:19:25.672973 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671076 4910 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:19:25.672973 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671078 4910 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:19:25.672973 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671081 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:19:25.672973 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671084 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:19:25.673497 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671087 4910 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:19:25.673497 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671089 4910 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:19:25.673497 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671094 4910 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:19:25.673497 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671096 4910 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:19:25.673497 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671099 4910 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:19:25.673497 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671102 4910 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:19:25.673497 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671104 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:19:25.673497 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671107 4910 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:19:25.673497 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671109 4910 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:19:25.673497 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671112 4910 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:19:25.673497 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671115 4910 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:19:25.673497 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671118 4910 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:19:25.673497 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671120 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:19:25.673497 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671123 4910 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:19:25.673497 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671125 4910 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:19:25.673497 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671128 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:19:25.673497 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671130 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:19:25.673497 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671133 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:19:25.673497 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671136 4910 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:19:25.673497 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671138 4910 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:19:25.674034 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671141 4910 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:19:25.674034 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671145 4910 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:19:25.674034 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671149 4910 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:19:25.674034 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671153 4910 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:19:25.674034 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671155 4910 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:19:25.674034 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671158 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:19:25.674034 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671161 4910 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:19:25.674034 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671163 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:19:25.674034 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671166 4910 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:19:25.674034 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671168 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:19:25.674034 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671171 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:19:25.674034 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671173 4910 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:19:25.674034 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671177 4910 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:19:25.674034 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671179 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:19:25.674034 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671183 4910 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:19:25.674034 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671187 4910 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:19:25.674034 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671193 4910 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:19:25.674034 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671196 4910 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:19:25.674034 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671199 4910 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:19:25.674507 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671202 4910 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:19:25.674507 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671204 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:19:25.674507 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671207 4910 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:19:25.674507 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671209 4910 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:19:25.674507 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671212 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:19:25.674507 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671214 4910 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:19:25.674507 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671217 4910 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:19:25.674507 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671219 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:19:25.674507 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671222 4910 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:19:25.674507 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671224 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:19:25.674507 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671227 4910 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:19:25.674507 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671230 4910 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:19:25.674507 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671232 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:19:25.674507 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671235 4910 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:19:25.674507 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671238 4910 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:19:25.674507 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671240 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:19:25.674507 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671243 4910 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:19:25.674507 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671246 4910 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:19:25.674507 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671248 4910 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:19:25.674999 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671251 4910 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:19:25.674999 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671253 4910 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:19:25.674999 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671256 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:19:25.674999 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671258 4910 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:19:25.674999 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671261 4910 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:19:25.674999 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671265 4910 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:19:25.674999 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671267 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:19:25.674999 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671270 4910 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:19:25.674999 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671274 4910 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:19:25.674999 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671277 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:19:25.674999 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671280 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:19:25.674999 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671282 4910 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:19:25.674999 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671285 4910 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:19:25.674999 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671287 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:19:25.674999 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671290 4910 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:19:25.674999 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671826 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:19:25.674999 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671834 4910 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:19:25.674999 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671837 4910 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:19:25.674999 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671840 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:19:25.674999 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671843 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:19:25.675500 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671846 4910 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:19:25.675500 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671849 4910 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:19:25.675500 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671851 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:19:25.675500 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671854 4910 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:19:25.675500 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671857 4910 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:19:25.675500 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671859 4910 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:19:25.675500 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671862 4910 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:19:25.675500 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671865 4910 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:19:25.675500 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671868 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:19:25.675500 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671871 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:19:25.675500 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671873 4910 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:19:25.675500 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671876 4910 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:19:25.675500 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671878 4910 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:19:25.675500 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671881 4910 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:19:25.675500 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671884 4910 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:19:25.675500 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671887 4910 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:19:25.675500 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671890 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:19:25.675500 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671893 4910 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:19:25.675500 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671896 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:19:25.675500 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671900 4910 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:19:25.676055 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671903 4910 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:19:25.676055 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671906 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:19:25.676055 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671920 4910 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:19:25.676055 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671923 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:19:25.676055 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671926 4910 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:19:25.676055 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671929 4910 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:19:25.676055 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671931 4910 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:19:25.676055 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671934 4910 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:19:25.676055 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671936 4910 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:19:25.676055 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671939 4910 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:19:25.676055 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671942 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:19:25.676055 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671944 4910 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:19:25.676055 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671947 4910 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:19:25.676055 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671951 4910 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:19:25.676055 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671954 4910 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:19:25.676055 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671957 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:19:25.676055 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671960 4910 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:19:25.676055 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671963 4910 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:19:25.676055 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671965 4910 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:19:25.676543 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671968 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:19:25.676543 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671970 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:19:25.676543 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671973 4910 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:19:25.676543 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671975 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:19:25.676543 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671978 4910 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:19:25.676543 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671981 4910 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:19:25.676543 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671983 4910 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:19:25.676543 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671986 4910 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:19:25.676543 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671988 4910 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:19:25.676543 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671991 4910 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:19:25.676543 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671994 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:19:25.676543 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671996 4910 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:19:25.676543 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.671999 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:19:25.676543 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672003 4910 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:19:25.676543 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672006 4910 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:19:25.676543 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672009 4910 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:19:25.676543 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672011 4910 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:19:25.676543 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672015 4910 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:19:25.676543 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672018 4910 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:19:25.676543 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672020 4910 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:19:25.677060 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672023 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:19:25.677060 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672025 4910 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:19:25.677060 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672028 4910 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:19:25.677060 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672030 4910 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:19:25.677060 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672033 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:19:25.677060 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672035 4910 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:19:25.677060 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672038 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:19:25.677060 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672041 4910 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:19:25.677060 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672043 4910 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:19:25.677060 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672046 4910 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:19:25.677060 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672048 4910 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:19:25.677060 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672050 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:19:25.677060 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672053 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:19:25.677060 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672056 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:19:25.677060 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672058 4910 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:19:25.677060 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672061 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:19:25.677060 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672063 4910 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:19:25.677060 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672066 4910 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:19:25.677060 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672068 4910 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:19:25.677060 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672071 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:19:25.677560 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672074 4910 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:19:25.677560 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672076 4910 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:19:25.677560 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672079 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:19:25.677560 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672082 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:19:25.677560 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672084 4910 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:19:25.677560 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672087 4910 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:19:25.677560 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672090 4910 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:19:25.677560 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672093 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:19:25.677560 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672095 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:19:25.677560 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672098 4910 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:19:25.677560 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672100 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:19:25.677560 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672103 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:19:25.677560 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672106 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:19:25.677560 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672108 4910 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:19:25.677560 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672111 4910 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:19:25.677560 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672113 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:19:25.677560 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672115 4910 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:19:25.677560 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672118 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:19:25.677560 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672121 4910 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:19:25.678129 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672126 4910 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:19:25.678129 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672129 4910 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:19:25.678129 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672132 4910 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:19:25.678129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672245 4910 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 19:19:25.678129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672263 4910 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 19:19:25.678129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672270 4910 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 19:19:25.678129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672275 4910 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 19:19:25.678129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672280 4910 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 19:19:25.678129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672283 4910 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 19:19:25.678129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672287 4910 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 19:19:25.678129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672291 4910 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 19:19:25.678129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672295 4910 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 19:19:25.678129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672298 4910 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 19:19:25.678129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672303 4910 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 19:19:25.678129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672309 4910 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 19:19:25.678129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672314 4910 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 19:19:25.678129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672318 4910 flags.go:64] FLAG: --cgroup-root="" Oct 31 19:19:25.678129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672322 4910 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 19:19:25.678129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672327 4910 flags.go:64] FLAG: --client-ca-file="" Oct 31 19:19:25.678129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672330 4910 flags.go:64] FLAG: --cloud-provider="external" Oct 31 19:19:25.678129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672334 4910 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 19:19:25.678129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672338 4910 flags.go:64] FLAG: --cluster-domain="" Oct 31 19:19:25.678129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672344 4910 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 19:19:25.679085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672347 4910 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 19:19:25.679085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672350 4910 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 19:19:25.679085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672354 4910 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 19:19:25.679085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672358 4910 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 19:19:25.679085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672361 4910 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 19:19:25.679085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672364 4910 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 19:19:25.679085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672368 4910 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 19:19:25.679085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672371 4910 flags.go:64] FLAG: --contention-profiling="false" Oct 31 19:19:25.679085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672374 4910 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 19:19:25.679085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672378 4910 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 19:19:25.679085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672381 4910 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 19:19:25.679085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672384 4910 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 19:19:25.679085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672389 4910 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 19:19:25.679085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672394 4910 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 19:19:25.679085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672410 4910 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 19:19:25.679085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672415 4910 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 19:19:25.679085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672420 4910 flags.go:64] FLAG: --enable-server="true" Oct 31 19:19:25.679085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672423 4910 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 19:19:25.679085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672429 4910 flags.go:64] FLAG: --event-burst="100" Oct 31 19:19:25.679085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672432 4910 flags.go:64] FLAG: --event-qps="50" Oct 31 19:19:25.679085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672435 4910 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 19:19:25.679085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672438 4910 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 19:19:25.679085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672442 4910 flags.go:64] FLAG: --eviction-hard="" Oct 31 19:19:25.679085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672446 4910 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 19:19:25.679767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672450 4910 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 19:19:25.679767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672453 4910 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 19:19:25.679767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672456 4910 flags.go:64] FLAG: --eviction-soft="" Oct 31 19:19:25.679767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672459 4910 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 19:19:25.679767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672462 4910 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 19:19:25.679767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672465 4910 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 19:19:25.679767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672469 4910 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 19:19:25.679767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672474 4910 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 19:19:25.679767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672477 4910 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 19:19:25.679767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672480 4910 flags.go:64] FLAG: --feature-gates="" Oct 31 19:19:25.679767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672487 4910 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 19:19:25.679767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672492 4910 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 19:19:25.679767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672497 4910 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 19:19:25.679767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672501 4910 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 19:19:25.679767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672504 4910 flags.go:64] FLAG: --healthz-port="10248" Oct 31 19:19:25.679767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672507 4910 flags.go:64] FLAG: --help="false" Oct 31 19:19:25.679767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672512 4910 flags.go:64] FLAG: --hostname-override="ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:19:25.679767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672515 4910 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 19:19:25.679767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672518 4910 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 19:19:25.679767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672521 4910 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 19:19:25.679767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672524 4910 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 19:19:25.679767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672529 4910 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 19:19:25.679767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672532 4910 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 19:19:25.680385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672535 4910 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 19:19:25.680385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672538 4910 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 19:19:25.680385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672541 4910 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 19:19:25.680385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672544 4910 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 19:19:25.680385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672547 4910 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 19:19:25.680385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672550 4910 flags.go:64] FLAG: --kube-reserved="" Oct 31 19:19:25.680385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672553 4910 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 19:19:25.680385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672556 4910 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 19:19:25.680385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672560 4910 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 19:19:25.680385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672564 4910 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 19:19:25.680385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672569 4910 flags.go:64] FLAG: --lock-file="" Oct 31 19:19:25.680385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672574 4910 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 19:19:25.680385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672579 4910 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 19:19:25.680385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672584 4910 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 19:19:25.680385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672590 4910 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 19:19:25.680385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672593 4910 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 19:19:25.680385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672599 4910 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 19:19:25.680385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672602 4910 flags.go:64] FLAG: --logging-format="text" Oct 31 19:19:25.680385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672605 4910 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 19:19:25.680385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672609 4910 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 19:19:25.680385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672611 4910 flags.go:64] FLAG: --manifest-url="" Oct 31 19:19:25.680385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672614 4910 flags.go:64] FLAG: --manifest-url-header="" Oct 31 19:19:25.680385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672619 4910 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 19:19:25.680385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672622 4910 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 19:19:25.680385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672626 4910 flags.go:64] FLAG: --max-pods="110" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672630 4910 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672634 4910 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672637 4910 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672640 4910 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672644 4910 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672649 4910 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672654 4910 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672665 4910 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672669 4910 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672672 4910 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672675 4910 flags.go:64] FLAG: --pod-cidr="" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672678 4910 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672680 4910 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672683 4910 flags.go:64] FLAG: --pods-per-core="0" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672687 4910 flags.go:64] FLAG: --port="10250" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672690 4910 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672693 4910 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0018c164098861af7" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672696 4910 flags.go:64] FLAG: --qos-reserved="" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672699 4910 flags.go:64] FLAG: --read-only-port="10255" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672702 4910 flags.go:64] FLAG: --register-node="true" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672705 4910 flags.go:64] FLAG: --register-with-taints="" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672709 4910 flags.go:64] FLAG: --registry-burst="10" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672712 4910 flags.go:64] FLAG: --registry-qps="5" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672716 4910 flags.go:64] FLAG: --reserved-cpus="" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672721 4910 flags.go:64] FLAG: --reserved-memory="" Oct 31 19:19:25.681063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672725 4910 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 19:19:25.681818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672728 4910 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 19:19:25.681818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672732 4910 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 19:19:25.681818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672735 4910 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 19:19:25.681818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672737 4910 flags.go:64] FLAG: --runonce="false" Oct 31 19:19:25.681818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672741 4910 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 19:19:25.681818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672744 4910 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 19:19:25.681818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672748 4910 flags.go:64] FLAG: --seccomp-default="false" Oct 31 19:19:25.681818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672751 4910 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 19:19:25.681818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672754 4910 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 19:19:25.681818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672758 4910 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 19:19:25.681818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672761 4910 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 19:19:25.681818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672764 4910 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 19:19:25.681818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672766 4910 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 19:19:25.681818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672769 4910 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 19:19:25.681818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672772 4910 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 19:19:25.681818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672775 4910 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 19:19:25.681818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672779 4910 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 19:19:25.681818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672782 4910 flags.go:64] FLAG: --system-cgroups="" Oct 31 19:19:25.681818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672785 4910 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 19:19:25.681818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672790 4910 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 19:19:25.681818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672794 4910 flags.go:64] FLAG: --tls-cert-file="" Oct 31 19:19:25.681818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672797 4910 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 19:19:25.681818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672801 4910 flags.go:64] FLAG: --tls-min-version="" Oct 31 19:19:25.681818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672804 4910 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 19:19:25.681818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672806 4910 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 19:19:25.682464 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672810 4910 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 19:19:25.682464 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672813 4910 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 19:19:25.682464 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672816 4910 flags.go:64] FLAG: --v="2" Oct 31 19:19:25.682464 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672820 4910 flags.go:64] FLAG: --version="false" Oct 31 19:19:25.682464 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672826 4910 flags.go:64] FLAG: --vmodule="" Oct 31 19:19:25.682464 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672830 4910 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 19:19:25.682464 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.672834 4910 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 19:19:25.682464 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672985 4910 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:19:25.682464 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672990 4910 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:19:25.682464 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672993 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:19:25.682464 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672996 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:19:25.682464 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.672999 4910 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:19:25.682464 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673002 4910 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:19:25.682464 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673008 4910 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:19:25.682464 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673010 4910 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:19:25.682464 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673013 4910 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:19:25.682464 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673016 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:19:25.682464 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673018 4910 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:19:25.682464 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673023 4910 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:19:25.682464 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673027 4910 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:19:25.682464 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673030 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:19:25.683191 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673033 4910 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:19:25.683191 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673037 4910 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:19:25.683191 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673040 4910 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:19:25.683191 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673042 4910 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:19:25.683191 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673046 4910 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:19:25.683191 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673050 4910 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:19:25.683191 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673053 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:19:25.683191 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673056 4910 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:19:25.683191 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673058 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:19:25.683191 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673061 4910 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:19:25.683191 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673064 4910 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:19:25.683191 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673066 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:19:25.683191 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673069 4910 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:19:25.683191 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673072 4910 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:19:25.683191 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673074 4910 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:19:25.683191 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673078 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:19:25.683191 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673081 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:19:25.683191 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673083 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:19:25.683873 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673086 4910 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:19:25.683873 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673089 4910 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:19:25.683873 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673092 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:19:25.683873 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673094 4910 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:19:25.683873 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673097 4910 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:19:25.683873 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673099 4910 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:19:25.683873 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673104 4910 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:19:25.683873 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673107 4910 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:19:25.683873 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673109 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:19:25.683873 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673112 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:19:25.683873 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673114 4910 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:19:25.683873 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673117 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:19:25.683873 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673120 4910 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:19:25.683873 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673123 4910 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:19:25.683873 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673125 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:19:25.683873 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673128 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:19:25.683873 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673130 4910 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:19:25.683873 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673133 4910 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:19:25.683873 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673135 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:19:25.683873 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673138 4910 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:19:25.684653 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673141 4910 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:19:25.684653 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673143 4910 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:19:25.684653 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673146 4910 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:19:25.684653 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673148 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:19:25.684653 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673151 4910 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:19:25.684653 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673153 4910 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:19:25.684653 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673156 4910 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:19:25.684653 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673159 4910 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:19:25.684653 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673161 4910 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:19:25.684653 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673165 4910 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:19:25.684653 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673168 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:19:25.684653 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673170 4910 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:19:25.684653 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673173 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:19:25.684653 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673176 4910 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:19:25.684653 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673179 4910 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:19:25.684653 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673182 4910 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:19:25.684653 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673185 4910 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:19:25.684653 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673187 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:19:25.684653 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673192 4910 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:19:25.685325 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673194 4910 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:19:25.685325 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673197 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:19:25.685325 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673199 4910 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:19:25.685325 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673202 4910 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:19:25.685325 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673205 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:19:25.685325 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673208 4910 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:19:25.685325 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673210 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:19:25.685325 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673213 4910 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:19:25.685325 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673215 4910 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:19:25.685325 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673218 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:19:25.685325 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673221 4910 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:19:25.685325 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673224 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:19:25.685325 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673226 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:19:25.685325 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673229 4910 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:19:25.685325 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673231 4910 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:19:25.685325 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673234 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:19:25.685325 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673237 4910 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:19:25.685325 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673239 4910 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:19:25.685325 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673242 4910 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:19:25.685325 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673244 4910 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:19:25.685325 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673247 4910 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:19:25.685984 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673249 4910 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:19:25.685984 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673253 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:19:25.685984 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673256 4910 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:19:25.685984 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673258 4910 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:19:25.685984 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673261 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:19:25.685984 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673264 4910 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:19:25.685984 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673267 4910 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:19:25.685984 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673269 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:19:25.685984 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673272 4910 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:19:25.685984 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673275 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:19:25.685984 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673278 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:19:25.685984 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673281 4910 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:19:25.685984 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673284 4910 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:19:25.685984 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673287 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:19:25.685984 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.673312 4910 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 19:19:25.685984 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673347 4910 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:19:25.686616 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673352 4910 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:19:25.686616 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673355 4910 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:19:25.686616 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673358 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:19:25.686616 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673361 4910 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:19:25.686616 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673364 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:19:25.686616 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673367 4910 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:19:25.686616 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673370 4910 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:19:25.686616 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673373 4910 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:19:25.686616 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673375 4910 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:19:25.686616 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673378 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:19:25.686616 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673381 4910 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:19:25.686616 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673383 4910 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:19:25.686616 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673386 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:19:25.686616 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673389 4910 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:19:25.686616 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673392 4910 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:19:25.686616 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673395 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:19:25.686616 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673397 4910 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:19:25.686616 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673400 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:19:25.686616 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673404 4910 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:19:25.686616 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673406 4910 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:19:25.687299 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673409 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:19:25.687299 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673412 4910 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:19:25.687299 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673415 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:19:25.687299 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673418 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:19:25.687299 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673421 4910 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:19:25.687299 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673423 4910 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:19:25.687299 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673427 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:19:25.687299 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673429 4910 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:19:25.687299 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673432 4910 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:19:25.687299 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673435 4910 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:19:25.687299 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673437 4910 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:19:25.687299 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673440 4910 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:19:25.687299 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673443 4910 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:19:25.687299 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673445 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:19:25.687299 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673448 4910 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:19:25.687299 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673450 4910 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:19:25.687299 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673453 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:19:25.687299 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673455 4910 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:19:25.687299 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673458 4910 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:19:25.687299 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673461 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:19:25.687849 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673463 4910 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:19:25.687849 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673466 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:19:25.687849 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673469 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:19:25.687849 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673471 4910 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:19:25.687849 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673474 4910 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:19:25.687849 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673476 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:19:25.687849 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673479 4910 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:19:25.687849 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673481 4910 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:19:25.687849 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673484 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:19:25.687849 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673487 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:19:25.687849 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673492 4910 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:19:25.687849 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673496 4910 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:19:25.687849 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673499 4910 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:19:25.687849 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673501 4910 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:19:25.687849 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673504 4910 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:19:25.687849 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673507 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:19:25.687849 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673510 4910 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:19:25.687849 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673514 4910 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:19:25.687849 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673519 4910 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:19:25.688388 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673522 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:19:25.688388 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673525 4910 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:19:25.688388 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673528 4910 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:19:25.688388 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673532 4910 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:19:25.688388 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673534 4910 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:19:25.688388 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673538 4910 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:19:25.688388 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673542 4910 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:19:25.688388 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673545 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:19:25.688388 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673547 4910 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:19:25.688388 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673550 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:19:25.688388 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673553 4910 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:19:25.688388 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673556 4910 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:19:25.688388 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673558 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:19:25.688388 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673561 4910 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:19:25.688388 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673564 4910 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:19:25.688388 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673566 4910 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:19:25.688388 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673569 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:19:25.688388 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673571 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:19:25.688822 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673574 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:19:25.688822 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673577 4910 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:19:25.688822 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673579 4910 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:19:25.688822 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673582 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:19:25.688822 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673584 4910 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:19:25.688822 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673588 4910 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:19:25.688822 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673591 4910 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:19:25.688822 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673594 4910 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:19:25.688822 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673597 4910 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:19:25.688822 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673600 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:19:25.688822 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673602 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:19:25.688822 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673605 4910 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:19:25.688822 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673608 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:19:25.688822 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673611 4910 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:19:25.688822 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673614 4910 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:19:25.688822 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673616 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:19:25.688822 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673619 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:19:25.688822 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673621 4910 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:19:25.688822 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673624 4910 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:19:25.688822 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673627 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:19:25.689350 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673629 4910 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:19:25.689350 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673632 4910 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:19:25.689350 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673634 4910 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:19:25.689350 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673637 4910 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:19:25.689350 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673640 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:19:25.689350 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673642 4910 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:19:25.689350 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673645 4910 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:19:25.689350 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.673648 4910 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:19:25.689350 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.677392 4910 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 19:19:25.689350 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.677410 4910 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 19:19:25.689350 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.677428 4910 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 19:19:25.689350 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.677432 4910 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 19:19:25.689350 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677491 4910 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:19:25.689350 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677497 4910 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:19:25.689350 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677501 4910 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:19:25.689350 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677504 4910 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:19:25.689350 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677506 4910 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:19:25.689350 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677509 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:19:25.689350 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677512 4910 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:19:25.689816 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677515 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:19:25.689816 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677518 4910 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:19:25.689816 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677520 4910 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:19:25.689816 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677523 4910 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:19:25.689816 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677526 4910 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:19:25.689816 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677529 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:19:25.689816 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677532 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:19:25.689816 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677535 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:19:25.689816 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677537 4910 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:19:25.689816 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677540 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:19:25.689816 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677543 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:19:25.689816 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677545 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:19:25.689816 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677548 4910 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:19:25.689816 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677551 4910 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:19:25.689816 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677554 4910 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:19:25.689816 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677557 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:19:25.689816 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677559 4910 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:19:25.689816 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677562 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:19:25.689816 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677565 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:19:25.689816 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677569 4910 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:19:25.690458 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677573 4910 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:19:25.690458 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677577 4910 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:19:25.690458 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677580 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:19:25.690458 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677583 4910 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:19:25.690458 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677586 4910 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:19:25.690458 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677589 4910 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:19:25.690458 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677591 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:19:25.690458 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677594 4910 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:19:25.690458 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677596 4910 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:19:25.690458 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677599 4910 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:19:25.690458 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677602 4910 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:19:25.690458 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677604 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:19:25.690458 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677607 4910 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:19:25.690458 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677610 4910 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:19:25.690458 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677612 4910 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:19:25.690458 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677615 4910 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:19:25.690458 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677617 4910 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:19:25.690458 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677620 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:19:25.690458 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677623 4910 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:19:25.690933 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677626 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:19:25.690933 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677628 4910 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:19:25.690933 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677643 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:19:25.690933 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677647 4910 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:19:25.690933 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677651 4910 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:19:25.690933 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677655 4910 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:19:25.690933 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677660 4910 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:19:25.690933 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677663 4910 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:19:25.690933 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677667 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:19:25.690933 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677671 4910 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:19:25.690933 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677675 4910 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:19:25.690933 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677680 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:19:25.690933 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677684 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:19:25.690933 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677686 4910 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:19:25.690933 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677689 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:19:25.690933 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677692 4910 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:19:25.690933 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677695 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:19:25.690933 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677697 4910 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:19:25.690933 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677700 4910 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:19:25.691403 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677702 4910 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:19:25.691403 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677705 4910 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:19:25.691403 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677707 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:19:25.691403 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677710 4910 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:19:25.691403 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677713 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:19:25.691403 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677716 4910 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:19:25.691403 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677719 4910 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:19:25.691403 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677721 4910 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:19:25.691403 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677724 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:19:25.691403 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677726 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:19:25.691403 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677729 4910 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:19:25.691403 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677731 4910 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:19:25.691403 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677734 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:19:25.691403 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677738 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:19:25.691403 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677741 4910 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:19:25.691403 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677744 4910 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:19:25.691403 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677748 4910 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:19:25.691403 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677752 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:19:25.691403 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677756 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:19:25.691403 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677759 4910 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:19:25.691875 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677762 4910 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:19:25.691875 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677764 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:19:25.691875 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677767 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:19:25.691875 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677769 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:19:25.691875 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677772 4910 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:19:25.691875 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677775 4910 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:19:25.691875 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677777 4910 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:19:25.691875 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677780 4910 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:19:25.691875 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677782 4910 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:19:25.691875 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677785 4910 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:19:25.691875 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677788 4910 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:19:25.691875 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677791 4910 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:19:25.691875 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677793 4910 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:19:25.691875 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677796 4910 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:19:25.691875 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677798 4910 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:19:25.691875 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677801 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:19:25.691875 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677803 4910 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:19:25.691875 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677806 4910 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:19:25.691875 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677808 4910 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:19:25.692358 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677811 4910 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:19:25.692358 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.677813 4910 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:19:25.692358 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.677840 4910 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 19:19:25.692358 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678729 4910 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:19:25.692358 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678755 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:19:25.692358 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678760 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:19:25.692358 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678765 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:19:25.692358 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678770 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:19:25.692358 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678774 4910 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:19:25.692358 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678778 4910 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:19:25.692358 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678784 4910 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:19:25.692358 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678788 4910 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:19:25.692358 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678792 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:19:25.692358 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678799 4910 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:19:25.692358 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678807 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:19:25.692723 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678818 4910 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:19:25.692723 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678823 4910 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:19:25.692723 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678828 4910 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:19:25.692723 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678832 4910 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:19:25.692723 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678837 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:19:25.692723 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678841 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:19:25.692723 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678845 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:19:25.692723 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678849 4910 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:19:25.692723 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678853 4910 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:19:25.692723 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678858 4910 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:19:25.692723 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678862 4910 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:19:25.692723 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678866 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:19:25.692723 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678870 4910 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:19:25.692723 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678879 4910 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:19:25.692723 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678883 4910 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:19:25.692723 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678887 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:19:25.692723 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678892 4910 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:19:25.692723 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678897 4910 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:19:25.692723 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678904 4910 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:19:25.692723 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678929 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:19:25.693234 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678934 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:19:25.693234 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678940 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:19:25.693234 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678945 4910 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:19:25.693234 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678949 4910 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:19:25.693234 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678953 4910 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:19:25.693234 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678957 4910 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:19:25.693234 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678967 4910 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:19:25.693234 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678971 4910 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:19:25.693234 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678976 4910 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:19:25.693234 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678980 4910 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:19:25.693234 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678984 4910 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:19:25.693234 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678988 4910 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:19:25.693234 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678992 4910 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:19:25.693234 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.678996 4910 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:19:25.693234 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679001 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:19:25.693234 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679005 4910 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:19:25.693234 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679018 4910 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:19:25.693234 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679023 4910 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:19:25.693234 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679032 4910 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:19:25.693737 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679036 4910 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:19:25.693737 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679041 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:19:25.693737 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679045 4910 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:19:25.693737 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679049 4910 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:19:25.693737 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679053 4910 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:19:25.693737 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679057 4910 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:19:25.693737 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679061 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:19:25.693737 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679065 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:19:25.693737 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679069 4910 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:19:25.693737 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679073 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:19:25.693737 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679143 4910 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:19:25.693737 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679282 4910 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:19:25.693737 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679297 4910 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:19:25.693737 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679301 4910 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:19:25.693737 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679304 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:19:25.693737 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679307 4910 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:19:25.693737 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679310 4910 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:19:25.693737 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679313 4910 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:19:25.693737 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679316 4910 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:19:25.693737 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679319 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:19:25.694241 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679324 4910 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:19:25.694241 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679327 4910 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:19:25.694241 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679330 4910 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:19:25.694241 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679333 4910 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:19:25.694241 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679336 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:19:25.694241 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679339 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:19:25.694241 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679342 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:19:25.694241 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679344 4910 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:19:25.694241 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679347 4910 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:19:25.694241 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679350 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:19:25.694241 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679353 4910 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:19:25.694241 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679355 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:19:25.694241 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679358 4910 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:19:25.694241 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679361 4910 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:19:25.694241 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679363 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:19:25.694241 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679366 4910 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:19:25.694241 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679369 4910 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:19:25.694241 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679371 4910 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:19:25.694241 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679374 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:19:25.694898 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679377 4910 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:19:25.694898 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679380 4910 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:19:25.694898 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679382 4910 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:19:25.694898 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679385 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:19:25.694898 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679388 4910 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:19:25.694898 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679391 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:19:25.694898 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679395 4910 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:19:25.694898 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679398 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:19:25.694898 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679400 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:19:25.694898 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679405 4910 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:19:25.694898 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679410 4910 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:19:25.694898 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679414 4910 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:19:25.694898 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679417 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:19:25.694898 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679422 4910 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:19:25.694898 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679425 4910 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:19:25.694898 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679427 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:19:25.695341 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.679463 4910 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 19:19:25.695341 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679507 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 19:19:25.695341 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679511 4910 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 19:19:25.695341 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679514 4910 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 19:19:25.695341 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679517 4910 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 19:19:25.695341 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679520 4910 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 19:19:25.695341 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679523 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 19:19:25.695341 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679527 4910 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 19:19:25.695341 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679531 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 19:19:25.695341 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679535 4910 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 19:19:25.695341 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679538 4910 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 19:19:25.695341 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679540 4910 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 19:19:25.695341 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679543 4910 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 19:19:25.695341 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679546 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 19:19:25.695341 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679548 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 19:19:25.695341 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679551 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 19:19:25.695742 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679553 4910 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 19:19:25.695742 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679556 4910 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 19:19:25.695742 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679559 4910 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 19:19:25.695742 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679561 4910 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 19:19:25.695742 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679564 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 19:19:25.695742 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679566 4910 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 19:19:25.695742 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679569 4910 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 19:19:25.695742 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679571 4910 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 19:19:25.695742 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679574 4910 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 19:19:25.695742 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679577 4910 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 19:19:25.695742 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679582 4910 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 19:19:25.695742 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679585 4910 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 19:19:25.695742 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679587 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 19:19:25.695742 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679590 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 19:19:25.695742 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679593 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 19:19:25.695742 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679596 4910 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 19:19:25.695742 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679599 4910 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 19:19:25.695742 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679601 4910 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 19:19:25.695742 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679604 4910 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 19:19:25.695742 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679606 4910 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 19:19:25.696521 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679609 4910 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 19:19:25.696521 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679611 4910 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 19:19:25.696521 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679614 4910 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 19:19:25.696521 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679617 4910 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 19:19:25.696521 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679619 4910 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 19:19:25.696521 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679622 4910 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 19:19:25.696521 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679625 4910 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 19:19:25.696521 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679627 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 19:19:25.696521 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679630 4910 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 19:19:25.696521 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679633 4910 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 19:19:25.696521 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679636 4910 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 19:19:25.696521 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679639 4910 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 19:19:25.696521 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679641 4910 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 19:19:25.696521 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679644 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 19:19:25.696521 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679647 4910 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 19:19:25.696521 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679649 4910 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 19:19:25.696521 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679652 4910 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 19:19:25.696521 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679654 4910 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 19:19:25.696521 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679657 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 19:19:25.696521 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679659 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 19:19:25.697312 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679662 4910 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 19:19:25.697312 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679664 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 19:19:25.697312 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679668 4910 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 19:19:25.697312 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679671 4910 feature_gate.go:386] unrecognized feature gate: Example Oct 31 19:19:25.697312 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679673 4910 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 19:19:25.697312 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679676 4910 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 19:19:25.697312 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679680 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 19:19:25.697312 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679683 4910 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 19:19:25.697312 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679685 4910 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 19:19:25.697312 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679688 4910 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 19:19:25.697312 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679691 4910 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 19:19:25.697312 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679693 4910 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 19:19:25.697312 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679696 4910 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 19:19:25.697312 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679698 4910 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 19:19:25.697312 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679701 4910 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 19:19:25.697312 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679705 4910 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 19:19:25.697312 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679708 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 19:19:25.697312 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679710 4910 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 19:19:25.697312 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679713 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 19:19:25.697823 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679715 4910 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 19:19:25.697823 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679718 4910 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 19:19:25.697823 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679721 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 19:19:25.697823 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679724 4910 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 19:19:25.697823 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679726 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 19:19:25.697823 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679729 4910 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 19:19:25.697823 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679731 4910 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 19:19:25.697823 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679734 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 19:19:25.697823 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679736 4910 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 19:19:25.697823 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679739 4910 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 19:19:25.697823 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679742 4910 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 19:19:25.697823 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679744 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 19:19:25.697823 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679747 4910 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 19:19:25.697823 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679749 4910 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 19:19:25.697823 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679752 4910 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 19:19:25.697823 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679756 4910 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 19:19:25.697823 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679759 4910 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 19:19:25.697823 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679761 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 19:19:25.697823 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679764 4910 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 19:19:25.698337 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679769 4910 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 19:19:25.698337 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679772 4910 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 19:19:25.698337 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679777 4910 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 19:19:25.698337 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679781 4910 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 19:19:25.698337 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679784 4910 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 19:19:25.698337 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679787 4910 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 19:19:25.698337 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679790 4910 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 19:19:25.698337 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679792 4910 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 19:19:25.698337 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679795 4910 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 19:19:25.698337 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679797 4910 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 19:19:25.698337 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679800 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 19:19:25.698337 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679803 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 19:19:25.698337 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:25.679805 4910 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 19:19:25.698337 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.679861 4910 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 19:19:25.698337 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.680879 4910 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 19:19:25.698337 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.680954 4910 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 19:19:25.698337 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.681138 4910 server.go:1014] "Starting client certificate rotation" Oct 31 19:19:25.698337 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.681285 4910 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 19:19:25.699106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.681344 4910 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 19:13:43 +0000 UTC" deadline="2028-06-12 14:23:36.951602511 +0000 UTC" Oct 31 19:19:25.699106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.681386 4910 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14155h4m11.270220103s" Oct 31 19:19:25.699106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.681598 4910 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 19:19:25.699106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.682658 4910 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 19:19:25.699106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.684087 4910 log.go:25] "Validated CRI v1 runtime API" Oct 31 19:19:25.699106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.685787 4910 log.go:25] "Validated CRI v1 image API" Oct 31 19:19:25.699106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.687207 4910 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 19:19:25.699106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.689066 4910 fs.go:135] Filesystem UUIDs: map[1a714163-8081-4292-a582-ab03cd37d2fc:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 91ac38ec-3f9c-4ad1-8eff-822e29738eb1:/dev/nvme0n1p3] Oct 31 19:19:25.699466 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.689086 4910 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a7f85425d882bdc757d28a51639f891ce7c284d793f8bfb07690ada43662c58/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a7f85425d882bdc757d28a51639f891ce7c284d793f8bfb07690ada43662c58/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19544c188d1e4a6b98188a935a654c1b0e9d94c776c707dc733952c54f0e2067/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19544c188d1e4a6b98188a935a654c1b0e9d94c776c707dc733952c54f0e2067/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1c6926b25843cf9433e0d70f29db2aed8179291a04d03e2f5add0e961325899f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1c6926b25843cf9433e0d70f29db2aed8179291a04d03e2f5add0e961325899f/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/208233d55fdbbef2a6b3722af35e78145ae9fa2022b91872c59d62678b9e4439/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/208233d55fdbbef2a6b3722af35e78145ae9fa2022b91872c59d62678b9e4439/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24cf43bbeef6d1006e432a1b0d9d36c4bcb020d1ecca07a90c945a69686c1097/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24cf43bbeef6d1006e432a1b0d9d36c4bcb020d1ecca07a90c945a69686c1097/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/336ffea7ff08409e19669798ffcc44ea0d354964196e08aafc954f9396d4ca05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/336ffea7ff08409e19669798ffcc44ea0d354964196e08aafc954f9396d4ca05/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/51292dc6a2394b3eccfbe90c1da601bf466be77c7db60e713d44b8d61403a6ff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/51292dc6a2394b3eccfbe90c1da601bf466be77c7db60e713d44b8d61403a6ff/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/58a3a53ca25510944bc534315dd2670e98583702f3bb6534584d79668465ad60/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/58a3a53ca25510944bc534315dd2670e98583702f3bb6534584d79668465ad60/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/69226f4df7c0decbee73876af4ddd5ed0647b62ae34de6e1094c79717bbcda3b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/69226f4df7c0decbee73876af4ddd5ed0647b62ae34de6e1094c79717bbcda3b/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6db78742cdd8d96d0c41b256e9d58eb82314da171eae09cb06fc9629394db69f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6db78742cdd8d96d0c41b256e9d58eb82314da171eae09cb06fc9629394db69f/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84fa1d57ccf00de8a4e885bb16678ee9716caaeaa5dfb50320f6d40c3ed5911c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84fa1d57ccf00de8a4e885bb16678ee9716caaeaa5dfb50320f6d40c3ed5911c/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/860bdd126bdb23ddbab4d48802a3be431f2d92a95f5d5716ba177928be8c2092/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/860bdd126bdb23ddbab4d48802a3be431f2d92a95f5d5716ba177928be8c2092/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a266a13bd4e51c27d828d0bb7e1adbb395a4f2faed8940c3282bc8fa68722f9f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a266a13bd4e51c27d828d0bb7e1adbb395a4f2faed8940c3282bc8fa68722f9f/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ada53789e04a83feca4c30151511cf86a4a6d7df24944c1c83b863ced5a2136c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ada53789e04a83feca4c30151511cf86a4a6d7df24944c1c83b863ced5a2136c/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1940201e2acbb0579e67e09eba8ea212ddb141274f55d9b7d9da554b57e149f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1940201e2acbb0579e67e09eba8ea212ddb141274f55d9b7d9da554b57e149f/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b9874207c5992fedc22950b400fa8eab1e93572799123f4c1b8a9a0085a9f2a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b9874207c5992fedc22950b400fa8eab1e93572799123f4c1b8a9a0085a9f2a1/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c54a9a71ab9d46f891569cbf6da57803dd50a5527f313b59c4b1353d218e696a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c54a9a71ab9d46f891569cbf6da57803dd50a5527f313b59c4b1353d218e696a/userdata/shm major:0 minor:305 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c94c48e8e9113d38dfd2e937b8021509e9458f0e33d347e220d5e27a246dbf95/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c94c48e8e9113d38dfd2e937b8021509e9458f0e33d347e220d5e27a246dbf95/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cb43e211e12624c829c6b1af0b205736d0fd4f1722fccd5a813ad06fe34e47ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cb43e211e12624c829c6b1af0b205736d0fd4f1722fccd5a813ad06fe34e47ba/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cdc79ea0e4f25f828ac0bf53e857206d62cee2ede1b798d52114280ccd5151fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cdc79ea0e4f25f828ac0bf53e857206d62cee2ede1b798d52114280ccd5151fe/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d47104b37f449b2e3b4f416da67e4be44ab266e8aff8244d4804ec228a4fce84/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d47104b37f449b2e3b4f416da67e4be44ab266e8aff8244d4804ec228a4fce84/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed192e0a5ee892c52cf2f4eefab17503ef6d0778bf98383753a44830616469b2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed192e0a5ee892c52cf2f4eefab17503ef6d0778bf98383753a44830616469b2/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f3b2be88f679fe1cf5dd30b6b6873254fe93aa6dcff09dbb01538f980e3642bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f3b2be88f679fe1cf5dd30b6b6873254fe93aa6dcff09dbb01538f980e3642bd/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1cc2220b-f30e-4879-b2cb-b2dd8e53b9f0/volumes/kubernetes.io~projected/kube-api-access-z6wxr:{mountpoint:/var/lib/kubelet/pods/1cc2220b-f30e-4879-b2cb-b2dd8e53b9f0/volumes/kubernetes.io~projected/kube-api-access-z6wxr major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d0b685c-252c-459c-9dba-d19c1bc6e5d0/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/1d0b685c-252c-459c-9dba-d19c1bc6e5d0/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/211efbea-29d0-4b2f-8c9e-6f3badcf77e5/volumes/kubernetes.io~projected/kube-api-access-9mk7v:{mountpoint:/var/lib/kubelet/pods/211efbea-29d0-4b2f-8c9e-6f3badcf77e5/volumes/kubernetes.io~projected/kube-api-access-9mk7v major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23a10416-51b1-415f-b2fc-892b4d662b1f/volumes/kubernetes.io~projected/kube-api-access-7djgh:{mountpoint:/var/lib/kubelet/pods/23a10416-51b1-415f-b2fc-892b4d662b1f/volumes/kubernetes.io~projected/kube-api-access-7djgh major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2552d079-24ff-4651-b77e-155af2fe48e9/volumes/kubernetes.io~projected/kube-api-access-t5mrm:{mountpoint:/var/lib/kubelet/pods/2552d079-24ff-4651-b77e-155af2fe48e9/volumes/kubernetes.io~projected/kube-api-access-t5mrm major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28c13795-a1de-4761-b3fc-feef76ac730d/volumes/kubernetes.io~projected/kube-api-access-bd4t7:{mountpoint:/var/lib/kubelet/pods/28c13795-a1de-4761-b3fc-feef76ac730d/volumes/kubernetes.io~projected/kube-api-access-bd4t7 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a063ed7-8ce0-414e-8bbf-7733c464b507/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/3a063ed7-8ce0-414e-8bbf-7733c464b507/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a063ed7-8ce0-414e-8bbf-7733c464b507/volumes/kubernetes.io~projected/kube-api-access-4zsbj:{mountpoint:/var/lib/kubelet/pods/3a063ed7-8ce0-414e-8bbf-7733c464b507/volumes/kubernetes.io~projected/kube-api-access-4zsbj major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a063ed7-8ce0-414e-8bbf-7733c464b507/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/3a063ed7-8ce0-414e-8bbf-7733c464b507/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a063ed7-8ce0-414e-8bbf-7733c464b507/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/3a063ed7-8ce0-414e-8bbf-7733c464b507/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a7f7d6d-958c-4973-9289-97f45526a431/volumes/kubernetes.io~projected/kube-api-access-q5trh:{mountpoint:/var/lib/kubelet/pods/3a7f7d6d-958c-4973-9289-97f45526a431/volumes/kubernetes.io~projected/kube-api-access-q5trh major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e84edf8-6e35-489b-b19b-17a5179e666e/volumes/kubernetes.io~projected/kube-api-access-g49vm:{mountpoint:/var/lib/kubelet/pods/3e84edf8-6e35-489b-b19b-17a5179e666e/volumes/kubernetes.io~projected/kube-api-access-g49vm major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47cff5a0-505b-45e9-9788-cbe4b557917e/volumes/kubernetes.io~projected/kube-api-access-8jzkh:{mountpoint:/var/lib/kubelet/pods/47cff5a0-505b-45e9-9788-cbe4b557917e/volumes/kubernetes.io~projected/kube-api-access-8jzkh major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4cd15118-0a59-4122-bdf1-56710c463262/volumes/kubernetes.io~projected/kube-api-access-nxpcv:{mountpoint:/var/lib/kubelet/pods/4cd15118-0a59-4122-bdf1-56710c463262/volumes/kubernetes.io~projected/kube-api-access-nxpcv major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ce261ba-08ff-418a-b3b4-b7e25e701bd8/volumes/kubernetes.io~projected/kube-api-access-4qtrj:{mountpoint:/var/lib/kubelet/pods/4ce261ba-08ff-418a-b3b4-b7e25e701bd8/volumes/kubernetes.io~projected/kube-api-access-4qtrj major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ce261ba-08ff-418a-b3b4-b7e25e701bd8/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/4ce261ba-08ff-418a-b3b4-b7e25e701bd8/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e729724-d310-4884-a2d0-54af5308124c/volumes/kubernetes.io~projected/kube-api-access-fn2tq:{mountpoint:/var/lib/kubelet/pods/4e729724-d310-4884-a2d0-54af5308124c/volumes/kubernetes.io~projected/kube-api-access-fn2tq major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52432f95-9eb4-4fcd-b795-e475f3399c62/volumes/kubernetes.io~projected/kube-api-access-pnwvq:{mountpoint:/var/lib/kubelet/pods/52432f95-9eb4-4fcd-b795-e475f3399c62/volumes/kubernetes.io~projected/kube-api-access-pnwvq major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5418e7b8-1471-4aa5-ad70-5d60b87c572a/volumes/kubernetes.io~projected/kube-api-access-vzwvn:{mountpoint:/var/lib/kubelet/pods/5418e7b8-1471-4aa5-ad70-5d60b87c572a/volumes/kubernetes.io~projected/kube-api-access-vzwvn major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5708be73-8282-4caa-bce3-46d4b708860e/volumes/kubernetes.io~projected/kube-api-access-9mjmg:{mountpoint:/var/lib/kubelet/pods/5708be73-8282-4caa-bce3-46d4b708860e/volumes/kubernetes.io~projected/kube-api-access-9mjmg major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/599fc0e5-05d8-46b0-90b3-60818a371161/volumes/kubernetes.io~projected/kube-api-access-b4lhh:{mountpoint:/var/lib/kubelet/pods/599fc0e5-05d8-46b0-90b3-60818a371161/volumes/kubernetes.io~projected/kube-api-access-b4lhh major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/599fc0e5-05d8-46b0-90b3-60818a371161/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/599fc0e5-05d8-46b0-90b3-60818a371161/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70c870bc-c9e6-4d7e-bc99-ba556ce31cde/volumes/kubernetes.io~projected/kube-api-access-4cj8g:{mountpoint:/var/lib/kubelet/pods/70c870bc-c9e6-4d7e-bc99-ba556ce31cde/volumes/kubernetes.io~projected/kube-api-access-4cj8g major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70c870bc-c9e6-4d7e-bc99-ba556ce31cde/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/70c870bc-c9e6-4d7e-bc99-ba556ce31cde/volumes/kubernetes.io~secret/default-certificate major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70c870bc-c9e6-4d7e-bc99-ba556ce31cde/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/70c870bc-c9e6-4d7e-bc99-ba556ce31cde/volumes/kubernetes.io~secret/stats-auth major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76e07820-d888-4138-a41a-59ff4565accd/volumes/kubernetes.io~projected/kube-api-access-dqspp:{mountpoint:/var/lib/kubelet/pods/76e07820-d888-4138-a41a-59ff4565accd/volumes/kubernetes.io~projected/kube-api-access-dqspp major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76e07820-d888-4138-a41a-59ff4565accd/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/76e07820-d888-4138-a41a-59ff4565accd/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bce545d-6640-49e8-848c-aa9f8db19902/volumes/kubernetes.io~projected/kube-api-access-l7fw4:{mountpoint:/var/lib/kubelet/pods/7bce545d-6640-49e8-848c-aa9f8db19902/volumes/kubernetes.io~projected/kube-api-access-l7fw4 major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81405623-de29-4fc6-bccf-447967669d89/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/81405623-de29-4fc6-bccf-447967669d89/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81405623-de29-4fc6-bccf-447967669d89/volumes/kubernetes.io~projected/kube-api-access-l7whr:{mountpoint:/var/lib/kubelet/pods/81405623-de29-4fc6-bccf-447967669d89/volumes/kubernetes.io~projected/kube-api-access-l7whr major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81405623-de29-4fc6-bccf-447967669d89/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/81405623-de29-4fc6-bccf-447967669d89/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/84f5548f-7d70-4391-89c9-3bd3c1784717/volumes/kubernetes.io~projected/kube-api-access-msgz5:{mountpoint:/var/lib/kubelet/pods/84f5548f-7d70-4391-89c9-3bd3c1784717/volumes/kubernetes.io~projected/kube-api-access-msgz5 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88a1fa4b-1191-434a-8e7c-783163ece24d/volumes/kubernetes.io~projected/kube-api-access-9q2cm:{mountpoint:/var/lib/kubelet/pods/88a1fa4b-1191-434a-8e7c-783163ece24d/volumes/kubernetes.io~projected/kube-api-access-9q2cm major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9255a1d5-2514-441e-b942-6c39843e38f8/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/9255a1d5-2514-441e-b942-6c39843e38f8/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92aaddbc-6268-443c-b3bd-9f74ab6b9a4b/volumes/kubernetes.io~projected/kube-api-access-rxnqt:{mountpoint:/var/lib/kubelet/pods/92aaddbc-6268-443c-b3bd-9f74ab6b9a4b/volumes/kubernetes.io~projected/kube-api-access-rxnqt major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92aaddbc-6268-443c-b3bd-9f74ab6b9a4b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/92aaddbc-6268-443c-b3bd-9f74ab6b9a4b/volumes/kubernetes.io~secret/cert major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f/volumes/kubernetes.io~projected/kube-api-access-gp54m:{mountpoint:/var/lib/kubelet/pods/ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f/volumes/kubernetes.io~projected/kube-api-access-gp54m major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf30e3e5-bfd9-4f93-9fe9-1514562c4267/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/bf30e3e5-bfd9-4f93-9fe9-1514562c4267/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf30e3e5-bfd9-4f93-9fe9-1514562c4267/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/bf30e3e5-bfd9-4f93-9fe9-1514562c4267/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf30e3e5-bfd9-4f93-9fe9-1514562c4267/volumes/kubernetes.io~projected/kube-api-access-b42xc:{mountpoint:/var/lib/kubelet/pods/bf30e3e5-bfd9-4f93-9fe9-1514562c4267/volumes/kubernetes.io~projected/kube-api-access-b42xc major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb1a9bea-399b-4281-9ada-a232790cc713/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/eb1a9bea-399b-4281-9ada-a232790cc713/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb1a9bea-399b-4281-9ada-a232790cc713/volumes/kubernetes.io~projected/kube-api-access-t4wnr:{mountpoint:/var/lib/kubelet/pods/eb1a9bea-399b-4281-9ada-a232790cc713/volumes/kubernetes.io~projected/kube-api-access-t4wnr major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb1a9bea-399b-4281-9ada-a232790cc713/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/eb1a9bea-399b-4281-9ada-a232790cc713/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb1a9bea-399b-4281-9ada-a232790cc713/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/eb1a9bea-399b-4281-9ada-a232790cc713/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e/volumes/kubernetes.io~projected/kube-api-access-w48sj:{mountpoint:/var/lib/kubelet/pods/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e/volumes/kubernetes.io~projected/kube-api-access-w48sj major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffb7495b-ef69-455c-99aa-22389d8c6d6e/volumes/kubernetes.io~projected/kube-api-access-9zkn5:{mountpoint:/var/lib/kubelet/pods/ffb7495b-ef69-455c-99aa-22389d8c6d6e/volumes/kubernetes.io~projected/kube-api-access-9zkn5 major:0 minor:212 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/77abe25fe317a4bbe8d3290a6eff66d87a9cb1cfcd7228c450a65a8da09a1d32/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/164721c614e51f2db2a5bd94f269f9a9cbcc30cf160b007a4d09114d1d33d762/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/e3c746e5244c6bbf0ec86c96e4f6486bdae4b40867b2072ccdc8ce3ecae0091e/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/189a70e6f41b94885f22b2e0033593608992d02b611e77441c2b3f5138aee8b8/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/0abafcd6cfcf00aa4fa9b50e91be36e5f52546940f9ca9c84c7cf892ff0e2c37/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/292d63cc831d10559564d040f4b65f0b7baef115de76b841ef84b85237f02666/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/f24ea03bf9dc0af7dc8ebe5c69cd09ea444d7365fa3080fcf0133d9f3f31764a/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/470e9fcc5568839d3134d6dd38f30b3d9f72ef9186b1ea33fe9749fac2d33cb5/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/136a827540310acc57b84bcedb4f3f69327d5b79950f160878d70d434c395e4e/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/5a4cd81a0ba1b882a92026b4c2545eaf79d8bedac4f0c513b25d84595b1d6c6c/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/c3e4f7a9893513e3eeb392de1c32641e72575ffe11ebff3af6dbff3e490b74da/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/d01fdcb83295cd314c3a6d30495c0a8fd33abcee104b2819c50633905d5dff65/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/3b4a07ea38f46f0464b9c9cce51556edce136ff9665247cd5280fda84c463259/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/b1dd4722dda85cc57ad0c1e4d1015536fb97e77f8943fb888d3d1273f730ed89/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/cf477029f84649dded6d6e792903c826c535b04cf6ac216662dc50f8e4c81c7e/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/94a014d189553c117d7e5826d74ecda976053e81eb8a07876e7dcf90d5031dbe/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/91e112bd1325f73fb2359c95116a52846814601203981e004268cac67325ba10/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/811150a226b9aafd0e3ed3d660d67ccffb7cd5d6d4016a0a312e5d068e163d87/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-178:{mountpoint:/var/lib/containers/storage/overlay/96b6b6c0ae01fc3d1c5f96e1d22f0bfd5c9be89541a9266953896a3e7c8c66b0/merged major:0 minor:178 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/3f07c14f5b507f8e2ef47802c26d6ab0521127ea2656883bd54c1901c2bb5c81/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/53e9ca5315e893e6d7ef30e5984b04ec0ac35e9d0b1ab93306ad1c4ab2a16e04/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/7a08f475785d3d3cd72fbe9e7dd6a2f36ada1bbe3b435c1d5745ba12c863716f/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/ab54b6e7a7125d17dc3e1272193297187ececef65a2e4604d7df78a7ee842703/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/07117b5213234012ee33c3ff008539fc860cffff4504ae24d2eb0a47d11500bb/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/05a47590d97c2a77c75c89328808a719521d967b07c3448c33f4f920d89bec68/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/ded9f77e200fdbb6b5211337e73c213036f47071d6d75fd12e4bcf2da6046fb1/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/6be21d1b235fa022d7b93b691c145c5d7b629d64467899bce3e2a73190552490/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/c9e247ce5782321da1d2b44c729de600eef0a5fe29eaf82cd5c8a3322fb9870c/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/a272d7e205c1f43c27d0ee460c1c63d3deb4d7f916c657d80171e1fb67c8a856/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/888c48fdf12639704d3cdc7cde48b1f523d3339beb0fcd13d185089b58b08d55/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/825f27ff412834d59dddcde47b543d465f8a06044ed4d9fd9433385bcf955f8e/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/f26ca0a3409b2f6e3cd2835434bd69ccb5252374e55f7700bcd401d7c9283028/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/f7a500d3abf37ea761d9190d809c859f0fd3d62341a2db3ae874d126235fe975/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/ba120470f45638e19d5ca4d45e0f1fb3fd79b267482ad2a0aeb206f9c09a51ed/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/e369047db7253abb43b95bf86c08859dbfa47b80602c7f2191c7dab393cbd45d/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/1a01a05b9c4bcb760beaa143c63d7d882a9969e0c4b222d6fb5bb36f3ea8a362/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/985c534b03071e0b37f594af55a3f0bb926743fe00753436b81a23b7eb6380d9/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/87dfcf685da69c6d94bea07276db0a3dc8bf613dc08498c60e4ba7e398f87870/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/e014c1bb6c515a770761c27e8ed274dd11fca0bc0da0135a06761080758639b8/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/edc5d1234161c648f5dfa6f681dc10c92b8344780c8973d5342cf9f41fc44e5a/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/a900fd36787942dc002b55ce18eefa74da356063cf72265b62f27db8230528a9/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/f1263c52a7c9d4e5a71bed785f6a465c81e2f88307f3eb12724dab90ba3bfaaf/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/5becac9ab4207cce00025d76730a0c3050cd371d0a59877d0d2ad9c1302cf524/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/12c8b591de8c39e7463e371298a88cc0cc2051c5fdefa3508cc9bd3063e3dc24/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/6c304aeed268b714a003f4d44773320ac3fecd448adaa39157269ad0735cb942/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/5cd056953f8356fbbcc82b7e586848b233acd7a873fb1c40c3d89562249aff93/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/defdb33cf58599c722ae4bf164ef1dd444bd8a8e9b61bd6c553109fa891e8678/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/f97c88558703af2f441077bda48a0752c4cda992031ad5c4bd52f581de321bbd/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/414227b125680a7db82667d250c3b6befa3937eab588ce361d5ec6f78451da39/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/b3842f915c661d0073f664f0671caa78d751e274d9c149d31157363ff0ae244c/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/7ca99ebe2432981babcb50f38c255d122575bf2a6354439b156dfe562d18fe72/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/8a4979b44f2d3d518d089ad8199f73cdce293032d458c72a696b2abf5880fa43/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/5f404f7e0f951bc38c1575540300c628336a95e525bf33bdcee038a10d9baa69/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/dd9eacfdc44f0b2852c17a418ee06599240899a8680b07a2d379c097e76edb35/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/f05e67781918807b1d185ca7fce4a4be18e54904081ef4de99192c4814f34c51/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/31a820635145a756f3eb1871f1b70af2c85176f64b9d378f15cefe418df50e04/merged major:0 minor:97 fsType:overlay blockSize:0}] Oct 31 19:19:25.699833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.697046 4910 manager.go:217] Machine: {Timestamp:2026-10-31 19:19:25.696243163 +0000 UTC m=+0.059063537 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096021 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2512f715f416bb51e08e0bfb77805a SystemUUID:ec2512f7-15f4-16bb-51e0-8e0bfb77805a BootID:9b66ad23-ebf1-4cea-bb7a-991c3f844cd3 Filesystems:[{Device:/var/lib/kubelet/pods/81405623-de29-4fc6-bccf-447967669d89/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/3e84edf8-6e35-489b-b19b-17a5179e666e/volumes/kubernetes.io~projected/kube-api-access-g49vm DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1cc2220b-f30e-4879-b2cb-b2dd8e53b9f0/volumes/kubernetes.io~projected/kube-api-access-z6wxr DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ada53789e04a83feca4c30151511cf86a4a6d7df24944c1c83b863ced5a2136c/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1c6926b25843cf9433e0d70f29db2aed8179291a04d03e2f5add0e961325899f/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/bf30e3e5-bfd9-4f93-9fe9-1514562c4267/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-178 DeviceMajor:0 DeviceMinor:178 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a7f85425d882bdc757d28a51639f891ce7c284d793f8bfb07690ada43662c58/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84fa1d57ccf00de8a4e885bb16678ee9716caaeaa5dfb50320f6d40c3ed5911c/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eb1a9bea-399b-4281-9ada-a232790cc713/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/599fc0e5-05d8-46b0-90b3-60818a371161/volumes/kubernetes.io~projected/kube-api-access-b4lhh DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/599fc0e5-05d8-46b0-90b3-60818a371161/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/76e07820-d888-4138-a41a-59ff4565accd/volumes/kubernetes.io~projected/kube-api-access-dqspp DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5708be73-8282-4caa-bce3-46d4b708860e/volumes/kubernetes.io~projected/kube-api-access-9mjmg DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/47cff5a0-505b-45e9-9788-cbe4b557917e/volumes/kubernetes.io~projected/kube-api-access-8jzkh DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a063ed7-8ce0-414e-8bbf-7733c464b507/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4cd15118-0a59-4122-bdf1-56710c463262/volumes/kubernetes.io~projected/kube-api-access-nxpcv DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3a7f7d6d-958c-4973-9289-97f45526a431/volumes/kubernetes.io~projected/kube-api-access-q5trh DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f3b2be88f679fe1cf5dd30b6b6873254fe93aa6dcff09dbb01538f980e3642bd/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9255a1d5-2514-441e-b942-6c39843e38f8/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/28c13795-a1de-4761-b3fc-feef76ac730d/volumes/kubernetes.io~projected/kube-api-access-bd4t7 DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb1a9bea-399b-4281-9ada-a232790cc713/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81405623-de29-4fc6-bccf-447967669d89/volumes/kubernetes.io~projected/kube-api-access-l7whr DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3a063ed7-8ce0-414e-8bbf-7733c464b507/volumes/kubernetes.io~projected/kube-api-access-4zsbj DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a266a13bd4e51c27d828d0bb7e1adbb395a4f2faed8940c3282bc8fa68722f9f/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed192e0a5ee892c52cf2f4eefab17503ef6d0778bf98383753a44830616469b2/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cb43e211e12624c829c6b1af0b205736d0fd4f1722fccd5a813ad06fe34e47ba/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3a063ed7-8ce0-414e-8bbf-7733c464b507/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e/volumes/kubernetes.io~projected/kube-api-access-w48sj DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/336ffea7ff08409e19669798ffcc44ea0d354964196e08aafc954f9396d4ca05/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf30e3e5-bfd9-4f93-9fe9-1514562c4267/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6db78742cdd8d96d0c41b256e9d58eb82314da171eae09cb06fc9629394db69f/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/88a1fa4b-1191-434a-8e7c-783163ece24d/volumes/kubernetes.io~projected/kube-api-access-9q2cm DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a063ed7-8ce0-414e-8bbf-7733c464b507/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b9874207c5992fedc22950b400fa8eab1e93572799123f4c1b8a9a0085a9f2a1/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70c870bc-c9e6-4d7e-bc99-ba556ce31cde/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5418e7b8-1471-4aa5-ad70-5d60b87c572a/volumes/kubernetes.io~projected/kube-api-access-vzwvn DeviceMajor:0 DeviceMinor:228 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e729724-d310-4884-a2d0-54af5308124c/volumes/kubernetes.io~projected/kube-api-access-fn2tq DeviceMajor:0 DeviceMinor:323 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/211efbea-29d0-4b2f-8c9e-6f3badcf77e5/volumes/kubernetes.io~projected/kube-api-access-9mk7v DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/81405623-de29-4fc6-bccf-447967669d89/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70c870bc-c9e6-4d7e-bc99-ba556ce31cde/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ffb7495b-ef69-455c-99aa-22389d8c6d6e/volumes/kubernetes.io~projected/kube-api-access-9zkn5 DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1940201e2acbb0579e67e09eba8ea212ddb141274f55d9b7d9da554b57e149f/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24cf43bbeef6d1006e432a1b0d9d36c4bcb020d1ecca07a90c945a69686c1097/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4ce261ba-08ff-418a-b3b4-b7e25e701bd8/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/76e07820-d888-4138-a41a-59ff4565accd/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eb1a9bea-399b-4281-9ada-a232790cc713/volumes/kubernetes.io~projected/kube-api-access-t4wnr DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/69226f4df7c0decbee73876af4ddd5ed0647b62ae34de6e1094c79717bbcda3b/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/58a3a53ca25510944bc534315dd2670e98583702f3bb6534584d79668465ad60/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/23a10416-51b1-415f-b2fc-892b4d662b1f/volumes/kubernetes.io~projected/kube-api-access-7djgh DeviceMajor:0 DeviceMinor:295 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/52432f95-9eb4-4fcd-b795-e475f3399c62/volumes/kubernetes.io~projected/kube-api-access-pnwvq DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19544c188d1e4a6b98188a935a654c1b0e9d94c776c707dc733952c54f0e2067/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/84f5548f-7d70-4391-89c9-3bd3c1784717/volumes/kubernetes.io~projected/kube-api-access-msgz5 DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1d0b685c-252c-459c-9dba-d19c1bc6e5d0/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2552d079-24ff-4651-b77e-155af2fe48e9/volumes/kubernetes.io~projected/kube-api-access-t5mrm DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cdc79ea0e4f25f828ac0bf53e857206d62cee2ede1b798d52114280ccd5151fe/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/51292dc6a2394b3eccfbe90c1da601bf466be77c7db60e713d44b8d61403a6ff/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70c870bc-c9e6-4d7e-bc99-ba556ce31cde/volumes/kubernetes.io~projected/kube-api-access-4cj8g DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb1a9bea-399b-4281-9ada-a232790cc713/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d47104b37f449b2e3b4f416da67e4be44ab266e8aff8244d4804ec228a4fce84/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c94c48e8e9113d38dfd2e937b8021509e9458f0e33d347e220d5e27a246dbf95/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/860bdd126bdb23ddbab4d48802a3be431f2d92a95f5d5716ba177928be8c2092/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4ce261ba-08ff-418a-b3b4-b7e25e701bd8/volumes/kubernetes.io~projected/kube-api-access-4qtrj DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/92aaddbc-6268-443c-b3bd-9f74ab6b9a4b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/92aaddbc-6268-443c-b3bd-9f74ab6b9a4b/volumes/kubernetes.io~projected/kube-api-access-rxnqt DeviceMajor:0 DeviceMinor:229 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf30e3e5-bfd9-4f93-9fe9-1514562c4267/volumes/kubernetes.io~projected/kube-api-access-b42xc DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/208233d55fdbbef2a6b3722af35e78145ae9fa2022b91872c59d62678b9e4439/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f/volumes/kubernetes.io~projected/kube-api-access-gp54m DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c54a9a71ab9d46f891569cbf6da57803dd50a5527f313b59c4b1353d218e696a/userdata/shm DeviceMajor:0 DeviceMinor:305 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7bce545d-6640-49e8-848c-aa9f8db19902/volumes/kubernetes.io~projected/kube-api-access-l7fw4 DeviceMajor:0 DeviceMinor:304 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0a7f85425d882bd MacAddress:1e:86:55:a0:f6:fe Speed:10000 Mtu:8901} {Name:19544c188d1e4a6 MacAddress:06:a4:67:ed:75:73 Speed:10000 Mtu:8901} {Name:1c6926b25843cf9 MacAddress:06:21:8f:79:43:d1 Speed:10000 Mtu:8901} {Name:336ffea7ff08409 MacAddress:de:2e:c4:1f:08:27 Speed:10000 Mtu:8901} {Name:58a3a53ca255109 MacAddress:fe:74:f6:22:a2:3b Speed:10000 Mtu:8901} {Name:69226f4df7c0dec MacAddress:be:b8:c8:24:14:5b Speed:10000 Mtu:8901} {Name:6db78742cdd8d96 MacAddress:52:b6:66:de:c8:dc Speed:10000 Mtu:8901} {Name:84fa1d57ccf00de MacAddress:3e:d9:a2:59:7e:27 Speed:10000 Mtu:8901} {Name:860bdd126bdb23d MacAddress:ee:62:d3:09:c2:3e Speed:10000 Mtu:8901} {Name:ada53789e04a83f MacAddress:26:a4:ef:82:79:45 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:db:82:d7:7d Speed:0 Mtu:9001} {Name:br-int MacAddress:02:ed:b7:71:08:aa Speed:0 Mtu:8901} {Name:c54a9a71ab9d46f MacAddress:16:ad:c5:bd:cf:b0 Speed:10000 Mtu:8901} {Name:d47104b37f449b2 MacAddress:e6:10:c2:59:d3:42 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:db:82:d7:7d Speed:0 Mtu:9001} {Name:f3b2be88f679fe1 MacAddress:6e:6c:c4:20:a1:7e Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:42:a9:50:d4:51:98 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:3a:03:3c:e6:c4:55 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 19:19:25.699833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.697611 4910 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 19:19:25.699833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.697672 4910 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 19:19:25.699833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.697961 4910 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 19:19:25.699833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.697980 4910 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-116.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 19:19:25.699833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.698105 4910 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 19:19:25.699833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.698112 4910 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 19:19:25.699833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.698125 4910 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 19:19:25.699833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.698143 4910 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 19:19:25.699833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.698163 4910 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 19:19:25.699833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.698306 4910 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 19:19:25.699833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.698405 4910 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 19:19:25.699833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.698439 4910 kubelet.go:493] "Attempting to sync node with API server" Oct 31 19:19:25.699833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.698451 4910 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 19:19:25.699833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.698461 4910 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 19:19:25.699833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.698470 4910 kubelet.go:405] "Adding apiserver pod source" Oct 31 19:19:25.699833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.698479 4910 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 19:19:25.699833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.699187 4910 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 19:19:25.699833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.699198 4910 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 19:19:25.699833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.699648 4910 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 19:19:25.701874 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.701855 4910 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 19:19:25.701874 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.701867 4910 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 19:19:25.702163 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.701978 4910 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 19:19:25.702163 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.702161 4910 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 19:19:25.702256 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.702188 4910 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 19:19:25.702308 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.702292 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 19:19:25.702308 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.702305 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 19:19:25.702380 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.702315 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 19:19:25.702380 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.702324 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 19:19:25.702380 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.702332 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 19:19:25.702380 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.702338 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 19:19:25.702380 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.702343 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 19:19:25.702380 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.702349 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 19:19:25.702380 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.702357 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 19:19:25.702380 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.702363 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 19:19:25.702380 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.702377 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 19:19:25.702742 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.702387 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 19:19:25.702742 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.702414 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 19:19:25.702742 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.702420 4910 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 19:19:25.706115 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.706101 4910 server.go:1263] "Started kubelet" Oct 31 19:19:25.706243 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.706192 4910 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 19:19:25.706302 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.706266 4910 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 19:19:25.706616 ip-10-0-142-116 systemd[1]: Started Kubernetes Kubelet. Oct 31 19:19:25.706771 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.706694 4910 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 19:19:25.706843 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.706762 4910 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 19:19:25.707748 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.707733 4910 server.go:317] "Adding debug handlers to kubelet server" Oct 31 19:19:25.710935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.710333 4910 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 19:19:25.710935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.710657 4910 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 19:19:25.710935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.710813 4910 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 19:19:25.710935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.710827 4910 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 19:19:25.711198 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.710978 4910 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 19:13:45 +0000 UTC" deadline="2028-06-30 18:56:43.4796363 +0000 UTC" Oct 31 19:19:25.711198 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.711018 4910 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14591h37m17.768622365s" Oct 31 19:19:25.711502 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.711476 4910 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 19:19:25.714437 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:25.714416 4910 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-116.us-east-2.compute.internal\" not found" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.716493 4910 factory.go:55] Registering systemd factory Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.716516 4910 factory.go:223] Registration of the systemd container factory successfully Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.716874 4910 factory.go:153] Registering CRI-O factory Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.716893 4910 factory.go:223] Registration of the crio container factory successfully Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.716960 4910 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.716980 4910 factory.go:103] Registering Raw factory Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.716996 4910 manager.go:1199] Started watching for new ooms in manager Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.717193 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a063ed7-8ce0-414e-8bbf-7733c464b507" volumeName="kubernetes.io/secret/3a063ed7-8ce0-414e-8bbf-7733c464b507-image-registry-private-configuration" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.717225 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47cff5a0-505b-45e9-9788-cbe4b557917e" volumeName="kubernetes.io/projected/47cff5a0-505b-45e9-9788-cbe4b557917e-kube-api-access-8jzkh" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.717241 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="599fc0e5-05d8-46b0-90b3-60818a371161" volumeName="kubernetes.io/configmap/599fc0e5-05d8-46b0-90b3-60818a371161-trusted-ca-bundle" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.717254 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="599fc0e5-05d8-46b0-90b3-60818a371161" volumeName="kubernetes.io/empty-dir/599fc0e5-05d8-46b0-90b3-60818a371161-snapshots" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.717267 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9255a1d5-2514-441e-b942-6c39843e38f8" volumeName="kubernetes.io/secret/9255a1d5-2514-441e-b942-6c39843e38f8-original-pull-secret" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.717279 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28c13795-a1de-4761-b3fc-feef76ac730d" volumeName="kubernetes.io/empty-dir/28c13795-a1de-4761-b3fc-feef76ac730d-tmp-dir" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.717291 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70c870bc-c9e6-4d7e-bc99-ba556ce31cde" volumeName="kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-stats-auth" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.717302 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84f5548f-7d70-4391-89c9-3bd3c1784717" volumeName="kubernetes.io/projected/84f5548f-7d70-4391-89c9-3bd3c1784717-kube-api-access-msgz5" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.717315 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4d13f0a-4f53-4659-9a4e-0e1020f98b2e" volumeName="kubernetes.io/configmap/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-config" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.717329 4910 manager.go:319] Starting recovery of all containers Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.717327 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4d13f0a-4f53-4659-9a4e-0e1020f98b2e" volumeName="kubernetes.io/projected/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-kube-api-access-w48sj" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721415 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47cff5a0-505b-45e9-9788-cbe4b557917e" volumeName="kubernetes.io/configmap/47cff5a0-505b-45e9-9788-cbe4b557917e-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721437 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5708be73-8282-4caa-bce3-46d4b708860e" volumeName="kubernetes.io/projected/5708be73-8282-4caa-bce3-46d4b708860e-kube-api-access-9mjmg" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721453 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f" volumeName="kubernetes.io/projected/ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f-kube-api-access-gp54m" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721467 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb1a9bea-399b-4281-9ada-a232790cc713" volumeName="kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-bound-sa-token" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721485 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e84edf8-6e35-489b-b19b-17a5179e666e" volumeName="kubernetes.io/projected/3e84edf8-6e35-489b-b19b-17a5179e666e-kube-api-access-g49vm" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721499 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ce261ba-08ff-418a-b3b4-b7e25e701bd8" volumeName="kubernetes.io/configmap/4ce261ba-08ff-418a-b3b4-b7e25e701bd8-config" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721513 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81405623-de29-4fc6-bccf-447967669d89" volumeName="kubernetes.io/configmap/81405623-de29-4fc6-bccf-447967669d89-env-overrides" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721525 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf30e3e5-bfd9-4f93-9fe9-1514562c4267" volumeName="kubernetes.io/empty-dir/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-tmp" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721554 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb1a9bea-399b-4281-9ada-a232790cc713" volumeName="kubernetes.io/configmap/eb1a9bea-399b-4281-9ada-a232790cc713-trusted-ca" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721565 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb1a9bea-399b-4281-9ada-a232790cc713" volumeName="kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-kube-api-access-t4wnr" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721577 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47cff5a0-505b-45e9-9788-cbe4b557917e" volumeName="kubernetes.io/configmap/47cff5a0-505b-45e9-9788-cbe4b557917e-cni-binary-copy" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721589 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5708be73-8282-4caa-bce3-46d4b708860e" volumeName="kubernetes.io/configmap/5708be73-8282-4caa-bce3-46d4b708860e-multus-daemon-config" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721605 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76e07820-d888-4138-a41a-59ff4565accd" volumeName="kubernetes.io/secret/76e07820-d888-4138-a41a-59ff4565accd-serving-cert" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721617 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92aaddbc-6268-443c-b3bd-9f74ab6b9a4b" volumeName="kubernetes.io/secret/92aaddbc-6268-443c-b3bd-9f74ab6b9a4b-cert" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721629 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb1a9bea-399b-4281-9ada-a232790cc713" volumeName="kubernetes.io/configmap/eb1a9bea-399b-4281-9ada-a232790cc713-registry-certificates" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721642 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="599fc0e5-05d8-46b0-90b3-60818a371161" volumeName="kubernetes.io/secret/599fc0e5-05d8-46b0-90b3-60818a371161-serving-cert" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721669 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76e07820-d888-4138-a41a-59ff4565accd" volumeName="kubernetes.io/configmap/76e07820-d888-4138-a41a-59ff4565accd-trusted-ca" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721682 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="84f5548f-7d70-4391-89c9-3bd3c1784717" volumeName="kubernetes.io/configmap/84f5548f-7d70-4391-89c9-3bd3c1784717-serviceca" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721701 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92aaddbc-6268-443c-b3bd-9f74ab6b9a4b" volumeName="kubernetes.io/projected/92aaddbc-6268-443c-b3bd-9f74ab6b9a4b-kube-api-access-rxnqt" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721719 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2552d079-24ff-4651-b77e-155af2fe48e9" volumeName="kubernetes.io/configmap/2552d079-24ff-4651-b77e-155af2fe48e9-iptables-alerter-script" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721731 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a063ed7-8ce0-414e-8bbf-7733c464b507" volumeName="kubernetes.io/configmap/3a063ed7-8ce0-414e-8bbf-7733c464b507-trusted-ca" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721744 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a063ed7-8ce0-414e-8bbf-7733c464b507" volumeName="kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-kube-api-access-4zsbj" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721756 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a063ed7-8ce0-414e-8bbf-7733c464b507" volumeName="kubernetes.io/secret/3a063ed7-8ce0-414e-8bbf-7733c464b507-installation-pull-secrets" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721769 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e729724-d310-4884-a2d0-54af5308124c" volumeName="kubernetes.io/empty-dir/4e729724-d310-4884-a2d0-54af5308124c-data-volume" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721781 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5418e7b8-1471-4aa5-ad70-5d60b87c572a" volumeName="kubernetes.io/empty-dir/5418e7b8-1471-4aa5-ad70-5d60b87c572a-tmp-dir" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721794 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81405623-de29-4fc6-bccf-447967669d89" volumeName="kubernetes.io/configmap/81405623-de29-4fc6-bccf-447967669d89-ovnkube-config" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721805 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f" volumeName="kubernetes.io/configmap/ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f-config" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721819 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f" volumeName="kubernetes.io/secret/ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f-serving-cert" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721832 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf30e3e5-bfd9-4f93-9fe9-1514562c4267" volumeName="kubernetes.io/empty-dir/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-tuned" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721845 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2552d079-24ff-4651-b77e-155af2fe48e9" volumeName="kubernetes.io/projected/2552d079-24ff-4651-b77e-155af2fe48e9-kube-api-access-t5mrm" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721857 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cf7d7f5-3936-45b3-ba2c-597cf99cfb89" volumeName="kubernetes.io/configmap/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-nginx-conf" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721871 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf30e3e5-bfd9-4f93-9fe9-1514562c4267" volumeName="kubernetes.io/projected/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-kube-api-access-b42xc" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721883 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb1a9bea-399b-4281-9ada-a232790cc713" volumeName="kubernetes.io/secret/eb1a9bea-399b-4281-9ada-a232790cc713-installation-pull-secrets" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721896 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="211efbea-29d0-4b2f-8c9e-6f3badcf77e5" volumeName="kubernetes.io/projected/211efbea-29d0-4b2f-8c9e-6f3badcf77e5-kube-api-access-9mk7v" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721932 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a063ed7-8ce0-414e-8bbf-7733c464b507" volumeName="kubernetes.io/empty-dir/3a063ed7-8ce0-414e-8bbf-7733c464b507-ca-trust-extracted" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721945 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cd15118-0a59-4122-bdf1-56710c463262" volumeName="kubernetes.io/projected/4cd15118-0a59-4122-bdf1-56710c463262-kube-api-access-nxpcv" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721958 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e729724-d310-4884-a2d0-54af5308124c" volumeName="kubernetes.io/projected/4e729724-d310-4884-a2d0-54af5308124c-kube-api-access-fn2tq" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721984 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52432f95-9eb4-4fcd-b795-e475f3399c62" volumeName="kubernetes.io/projected/52432f95-9eb4-4fcd-b795-e475f3399c62-kube-api-access-pnwvq" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.721996 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5418e7b8-1471-4aa5-ad70-5d60b87c572a" volumeName="kubernetes.io/configmap/5418e7b8-1471-4aa5-ad70-5d60b87c572a-config-volume" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722008 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5708be73-8282-4caa-bce3-46d4b708860e" volumeName="kubernetes.io/configmap/5708be73-8282-4caa-bce3-46d4b708860e-cni-binary-copy" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722020 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76e07820-d888-4138-a41a-59ff4565accd" volumeName="kubernetes.io/projected/76e07820-d888-4138-a41a-59ff4565accd-kube-api-access-dqspp" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722034 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a063ed7-8ce0-414e-8bbf-7733c464b507" volumeName="kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-bound-sa-token" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722047 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d0b685c-252c-459c-9dba-d19c1bc6e5d0" volumeName="kubernetes.io/configmap/1d0b685c-252c-459c-9dba-d19c1bc6e5d0-konnectivity-ca" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722060 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a063ed7-8ce0-414e-8bbf-7733c464b507" volumeName="kubernetes.io/configmap/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-certificates" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722072 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5418e7b8-1471-4aa5-ad70-5d60b87c572a" volumeName="kubernetes.io/projected/5418e7b8-1471-4aa5-ad70-5d60b87c572a-kube-api-access-vzwvn" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722085 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="599fc0e5-05d8-46b0-90b3-60818a371161" volumeName="kubernetes.io/projected/599fc0e5-05d8-46b0-90b3-60818a371161-kube-api-access-b4lhh" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722106 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88a1fa4b-1191-434a-8e7c-783163ece24d" volumeName="kubernetes.io/projected/88a1fa4b-1191-434a-8e7c-783163ece24d-kube-api-access-9q2cm" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722119 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28c13795-a1de-4761-b3fc-feef76ac730d" volumeName="kubernetes.io/projected/28c13795-a1de-4761-b3fc-feef76ac730d-kube-api-access-bd4t7" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722133 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a7f7d6d-958c-4973-9289-97f45526a431" volumeName="kubernetes.io/projected/3a7f7d6d-958c-4973-9289-97f45526a431-kube-api-access-q5trh" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722145 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cd15118-0a59-4122-bdf1-56710c463262" volumeName="kubernetes.io/configmap/4cd15118-0a59-4122-bdf1-56710c463262-telemetry-config" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722159 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70c870bc-c9e6-4d7e-bc99-ba556ce31cde" volumeName="kubernetes.io/projected/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-kube-api-access-4cj8g" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722172 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76e07820-d888-4138-a41a-59ff4565accd" volumeName="kubernetes.io/configmap/76e07820-d888-4138-a41a-59ff4565accd-config" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722185 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1cc2220b-f30e-4879-b2cb-b2dd8e53b9f0" volumeName="kubernetes.io/projected/1cc2220b-f30e-4879-b2cb-b2dd8e53b9f0-kube-api-access-z6wxr" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722197 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47cff5a0-505b-45e9-9788-cbe4b557917e" volumeName="kubernetes.io/configmap/47cff5a0-505b-45e9-9788-cbe4b557917e-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722210 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="599fc0e5-05d8-46b0-90b3-60818a371161" volumeName="kubernetes.io/configmap/599fc0e5-05d8-46b0-90b3-60818a371161-service-ca-bundle" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722222 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70c870bc-c9e6-4d7e-bc99-ba556ce31cde" volumeName="kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-default-certificate" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722235 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bce545d-6640-49e8-848c-aa9f8db19902" volumeName="kubernetes.io/projected/7bce545d-6640-49e8-848c-aa9f8db19902-kube-api-access-l7fw4" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722246 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ce261ba-08ff-418a-b3b4-b7e25e701bd8" volumeName="kubernetes.io/secret/4ce261ba-08ff-418a-b3b4-b7e25e701bd8-serving-cert" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722258 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81405623-de29-4fc6-bccf-447967669d89" volumeName="kubernetes.io/projected/81405623-de29-4fc6-bccf-447967669d89-kube-api-access-l7whr" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722271 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d0b685c-252c-459c-9dba-d19c1bc6e5d0" volumeName="kubernetes.io/secret/1d0b685c-252c-459c-9dba-d19c1bc6e5d0-agent-certs" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722286 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23a10416-51b1-415f-b2fc-892b4d662b1f" volumeName="kubernetes.io/projected/23a10416-51b1-415f-b2fc-892b4d662b1f-kube-api-access-7djgh" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722297 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ce261ba-08ff-418a-b3b4-b7e25e701bd8" volumeName="kubernetes.io/projected/4ce261ba-08ff-418a-b3b4-b7e25e701bd8-kube-api-access-4qtrj" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722309 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81405623-de29-4fc6-bccf-447967669d89" volumeName="kubernetes.io/configmap/81405623-de29-4fc6-bccf-447967669d89-ovnkube-script-lib" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722320 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81405623-de29-4fc6-bccf-447967669d89" volumeName="kubernetes.io/secret/81405623-de29-4fc6-bccf-447967669d89-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722333 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb1a9bea-399b-4281-9ada-a232790cc713" volumeName="kubernetes.io/empty-dir/eb1a9bea-399b-4281-9ada-a232790cc713-ca-trust-extracted" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722346 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb1a9bea-399b-4281-9ada-a232790cc713" volumeName="kubernetes.io/secret/eb1a9bea-399b-4281-9ada-a232790cc713-image-registry-private-configuration" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722361 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="599fc0e5-05d8-46b0-90b3-60818a371161" volumeName="kubernetes.io/empty-dir/599fc0e5-05d8-46b0-90b3-60818a371161-tmp" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722373 4910 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb7495b-ef69-455c-99aa-22389d8c6d6e" volumeName="kubernetes.io/projected/ffb7495b-ef69-455c-99aa-22389d8c6d6e-kube-api-access-9zkn5" seLinuxMountContext="" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722385 4910 reconstruct.go:97] "Volume reconstruction finished" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.722394 4910 reconciler.go:29] "Reconciler: start to sync state" Oct 31 19:19:25.724879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.724476 4910 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:19:25.729140 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.725762 4910 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:19:25.732093 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.732049 4910 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:19:25.732922 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.732886 4910 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 19:19:25.735636 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.735617 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/1.log" Oct 31 19:19:25.737508 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.737490 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/1.log" Oct 31 19:19:25.744790 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.744645 4910 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 19:19:25.746865 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.746372 4910 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 19:19:25.746865 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.746393 4910 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 19:19:25.746865 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.746417 4910 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 19:19:25.746865 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:25.746457 4910 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 19:19:25.757524 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.757500 4910 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 19:19:25.764695 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.764644 4910 generic.go:356] "Generic (PLEG): container finished" podID="2993240a68b51a645ff6b733bc57689d" containerID="322797fac2c246960ad190be1a22df3a0351675e20897170d6b3448082e77c96" exitCode=0 Oct 31 19:19:25.773231 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.773170 4910 generic.go:356] "Generic (PLEG): container finished" podID="47cff5a0-505b-45e9-9788-cbe4b557917e" containerID="1364f09091e32aec947d72de9eed8b75eb4f5ab25cc5abaf4999a1c7f58bd7e3" exitCode=0 Oct 31 19:19:25.773231 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.773195 4910 generic.go:356] "Generic (PLEG): container finished" podID="47cff5a0-505b-45e9-9788-cbe4b557917e" containerID="94d29bf03388cbe8d79beb93341418e8ccfd1d6bfeb09089dec603d19dbff25b" exitCode=0 Oct 31 19:19:25.773231 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.773206 4910 generic.go:356] "Generic (PLEG): container finished" podID="47cff5a0-505b-45e9-9788-cbe4b557917e" containerID="bfaa565d5f19ee223e2a41a44bcce5d1fc2baf3a38e87e9b32a1954d19d107b5" exitCode=0 Oct 31 19:19:25.773231 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.773214 4910 generic.go:356] "Generic (PLEG): container finished" podID="47cff5a0-505b-45e9-9788-cbe4b557917e" containerID="d69a8fa8428e8bc841753a2ce0f3990f2929a6f508f726cb41967e9ed29fb59b" exitCode=0 Oct 31 19:19:25.773231 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.773223 4910 generic.go:356] "Generic (PLEG): container finished" podID="47cff5a0-505b-45e9-9788-cbe4b557917e" containerID="ec22f0135a308db2eb1de7e2573b618d4858fb4b9b9647e9cd0981ec1e94b62a" exitCode=0 Oct 31 19:19:25.773231 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.773231 4910 generic.go:356] "Generic (PLEG): container finished" podID="47cff5a0-505b-45e9-9788-cbe4b557917e" containerID="7cb2fefa3e14f2c331683970309db8de56404d20b534f9e32e7029a6569daec1" exitCode=0 Oct 31 19:19:25.800559 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.800536 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/1.log" Oct 31 19:19:25.800676 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.800564 4910 generic.go:356] "Generic (PLEG): container finished" podID="76e07820-d888-4138-a41a-59ff4565accd" containerID="3a9fb18642a3e4ea0da071d9b3009cfddbd0e22c650b9a07b4a1d921bd019368" exitCode=255 Oct 31 19:19:25.813245 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.813221 4910 manager.go:324] Recovery completed Oct 31 19:19:25.832748 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.832722 4910 cpu_manager.go:226] "Starting" policy="none" Oct 31 19:19:25.832748 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.832737 4910 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 19:19:25.832880 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.832755 4910 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 19:19:25.832925 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.832875 4910 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 19:19:25.832925 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.832886 4910 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 19:19:25.832925 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.832903 4910 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 19:19:25.832925 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.832923 4910 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 19:19:25.833050 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.832929 4910 policy_none.go:50] "Start" Oct 31 19:19:25.833050 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.832948 4910 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 19:19:25.833050 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.832960 4910 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 19:19:25.833125 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.833097 4910 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 19:19:25.833125 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.833105 4910 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 19:19:25.833125 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.833110 4910 policy_none.go:44] "Start" Oct 31 19:19:25.837294 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.837264 4910 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 19:19:25.840483 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.840471 4910 manager.go:341] "Starting Device Plugin manager" Oct 31 19:19:25.841031 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:25.840497 4910 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 19:19:25.841031 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.841024 4910 server.go:87] "Starting device plugin registration server" Oct 31 19:19:25.841269 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.841260 4910 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 19:19:25.841303 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.841270 4910 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 19:19:25.841391 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.841368 4910 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 19:19:25.841523 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.841479 4910 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 19:19:25.841611 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.841532 4910 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 19:19:25.841611 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.841539 4910 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 19:19:25.841707 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.841593 4910 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T19:19:25.841512636Z","UUID":"35219b4d-373d-4d42-84f4-1badf645f0f4","Handler":null,"Name":"","Endpoint":""} Oct 31 19:19:25.843225 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:25.843145 4910 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 19:19:25.843703 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.843682 4910 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 19:19:25.843801 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.843723 4910 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 19:19:25.846610 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.846573 4910 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal"] Oct 31 19:19:25.847608 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.847590 4910 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:19:25.847689 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.847603 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal" event={"ID":"956815bb47502a7ffa36d27c2bef7876","Type":"ContainerStarted","Data":"949ec3e3746b843fe26e624b691d3e5fca870d1d89d8c23e6ead0b6a1317bf41"} Oct 31 19:19:25.847689 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.847646 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal" event={"ID":"956815bb47502a7ffa36d27c2bef7876","Type":"ContainerStarted","Data":"b1940201e2acbb0579e67e09eba8ea212ddb141274f55d9b7d9da554b57e149f"} Oct 31 19:19:25.847689 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.847660 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" event={"ID":"2993240a68b51a645ff6b733bc57689d","Type":"ContainerStarted","Data":"9ae0dc728cfb6d94d801453920d29f33e7bfbf6f49895960c605cf0774740d74"} Oct 31 19:19:25.847689 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.847672 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" event={"ID":"2993240a68b51a645ff6b733bc57689d","Type":"ContainerDied","Data":"322797fac2c246960ad190be1a22df3a0351675e20897170d6b3448082e77c96"} Oct 31 19:19:25.847689 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.847685 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" event={"ID":"2993240a68b51a645ff6b733bc57689d","Type":"ContainerStarted","Data":"a266a13bd4e51c27d828d0bb7e1adbb395a4f2faed8940c3282bc8fa68722f9f"} Oct 31 19:19:25.847941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.847900 4910 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:19:25.870789 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.870739 4910 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 19:19:25.870897 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:25.870834 4910 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:19:25.872289 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.872273 4910 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 19:19:25.872364 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:25.872314 4910 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:19:25.934050 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.934016 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2993240a68b51a645ff6b733bc57689d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal\" (UID: \"2993240a68b51a645ff6b733bc57689d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:19:25.934050 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.934049 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2993240a68b51a645ff6b733bc57689d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal\" (UID: \"2993240a68b51a645ff6b733bc57689d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:19:25.934225 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.934068 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/956815bb47502a7ffa36d27c2bef7876-config\") pod \"kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal\" (UID: \"956815bb47502a7ffa36d27c2bef7876\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:19:25.942114 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.942092 4910 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 19:19:25.943523 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.943502 4910 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 19:19:25.943622 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.943532 4910 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 19:19:25.943622 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.943547 4910 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-116.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 19:19:25.943622 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.943592 4910 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:19:25.961017 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.960997 4910 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:19:25.961155 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:25.961069 4910 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:19:26.034865 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.034781 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2993240a68b51a645ff6b733bc57689d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal\" (UID: \"2993240a68b51a645ff6b733bc57689d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:19:26.034865 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.034818 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2993240a68b51a645ff6b733bc57689d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal\" (UID: \"2993240a68b51a645ff6b733bc57689d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:19:26.034865 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.034840 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/956815bb47502a7ffa36d27c2bef7876-config\") pod \"kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal\" (UID: \"956815bb47502a7ffa36d27c2bef7876\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:19:26.035102 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.034920 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2993240a68b51a645ff6b733bc57689d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal\" (UID: \"2993240a68b51a645ff6b733bc57689d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:19:26.035102 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.034957 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2993240a68b51a645ff6b733bc57689d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal\" (UID: \"2993240a68b51a645ff6b733bc57689d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:19:26.035102 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.034907 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/956815bb47502a7ffa36d27c2bef7876-config\") pod \"kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal\" (UID: \"956815bb47502a7ffa36d27c2bef7876\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:19:26.699324 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.699287 4910 apiserver.go:52] "Watching apiserver" Oct 31 19:19:26.715813 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.715783 4910 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 19:19:26.716468 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.716440 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgrs8","openshift-network-operator/iptables-alerter-qn24x","openshift-kube-storage-version-migrator/migrator-66df5d496b-4fsp7","openshift-network-console/networking-console-plugin-86c478bb88-28mzh","openshift-insights/insights-operator-7c4986d449-mtmp6","openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n","openshift-image-registry/node-ca-8m482","openshift-network-diagnostics/network-check-target-5mkxz","openshift-image-registry/image-registry-6f878896cc-hzn9v","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4","openshift-image-registry/image-registry-5c8ccd4bc6-xlml9","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm","openshift-multus/network-metrics-daemon-dfbrp","openshift-dns/node-resolver-85mq2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal","kube-system/konnectivity-agent-wxdtf","kube-system/global-pull-secret-syncer-lhtkb","kube-system/kas-connection-checker-685895c7c9-vszkf","openshift-dns/dns-default-5nm2g","openshift-ingress/router-default-8c75d8d7f-tzzdl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7","openshift-ingress-canary/ingress-canary-q8j58","openshift-multus/multus-additional-cni-plugins-bhqht","kube-system/kas-connection-checker-685895c7c9-4mdb2","kube-system/kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal","openshift-network-diagnostics/network-check-source-6f78d59f99-l84pf","kube-system/kas-connection-checker-685895c7c9-xmfjb","openshift-multus/multus-tm5fh","openshift-console-operator/console-operator-7cd56b8dbc-6vpwc","openshift-insights/insights-runtime-extractor-5nkwp","openshift-ovn-kubernetes/ovnkube-node-m2pjd","openshift-cluster-node-tuning-operator/tuned-4ngtp"] Oct 31 19:19:26.718715 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.718691 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" Oct 31 19:19:26.718990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.718946 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:26.719663 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.719648 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:26.720034 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.720019 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:26.720842 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.720828 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-28mzh" Oct 31 19:19:26.726465 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.726450 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:26.727408 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.727383 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:19:26.728470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.728453 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:26.728684 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.728663 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 19:19:26.730065 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.730035 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 19:19:26.730190 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.730167 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 19:19:26.730467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.730451 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 19:19:26.730596 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.730583 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 19:19:26.730648 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.730609 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 19:19:26.730691 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.730648 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 19:19:26.730736 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.730729 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 19:19:26.730858 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.730844 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 19:19:26.731298 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.731277 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:26.735430 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.735407 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 19:19:26.735530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.735514 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 19:19:26.735586 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.735516 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 19:19:26.735586 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.735570 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 19:19:26.735676 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.735621 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 19:19:26.735676 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.735620 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 19:19:26.735770 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.735702 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 19:19:26.735770 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.735714 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 19:19:26.735770 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.735717 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 19:19:26.735921 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.735772 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 19:19:26.735921 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.735778 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 19:19:26.735921 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.735789 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 19:19:26.735921 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.735791 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 19:19:26.735921 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.735805 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 19:19:26.735921 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.735791 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 19:19:26.735921 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.735813 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 19:19:26.735921 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.735830 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 19:19:26.735921 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.735873 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 19:19:26.735921 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.735890 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 19:19:26.737052 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.737040 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 19:19:26.737375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.737364 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 19:19:26.737431 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.737412 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 19:19:26.737490 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.737444 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 19:19:26.737490 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.737453 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 19:19:26.737490 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.737472 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 19:19:26.737490 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.737478 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 19:19:26.737490 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.737429 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 19:19:26.737709 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.737432 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 19:19:26.737709 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.737471 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 19:19:26.737709 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.737512 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 19:19:26.737709 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.737517 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 19:19:26.737709 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.737518 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 19:19:26.737709 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.737445 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 19:19:26.737709 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.737479 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 19:19:26.737709 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.737520 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 19:19:26.737709 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.737564 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 19:19:26.738948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.738931 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 19:19:26.738948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.738938 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 19:19:26.739324 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.739307 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 19:19:26.739411 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.739354 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 19:19:26.739451 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.739375 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 19:19:26.739451 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.739378 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 19:19:26.741086 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.741069 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 19:19:26.741152 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.741085 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 19:19:26.741152 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.741116 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 19:19:26.741313 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.741299 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 19:19:26.741358 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.741320 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 19:19:26.741358 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.741335 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 19:19:26.741358 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.741320 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 19:19:26.742055 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.742043 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 19:19:26.744948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.744932 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 19:19:26.745116 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.745087 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 19:19:26.746772 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.746752 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 19:19:26.746858 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.746770 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 19:19:26.746858 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.746783 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 19:19:26.746858 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.746814 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 19:19:26.746858 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.746817 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 19:19:26.746858 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.746830 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 19:19:26.748749 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.748733 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 19:19:26.748810 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.748741 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 19:19:26.748810 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.748772 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 19:19:26.750668 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.750649 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 19:19:26.750745 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.750695 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 19:19:26.750745 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.750724 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mxmqk\"" Oct 31 19:19:26.750745 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.750730 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gfbm\"" Oct 31 19:19:26.750897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.750746 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-w5n5s\"" Oct 31 19:19:26.750897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.750758 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nngmt\"" Oct 31 19:19:26.750897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.750764 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mpqbj\"" Oct 31 19:19:26.750897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.750761 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kbgxr\"" Oct 31 19:19:26.750897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.750782 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 19:19:26.750897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.750793 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zwg2q\"" Oct 31 19:19:26.750897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.750779 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dcmmt\"" Oct 31 19:19:26.750897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.750837 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jd7hq\"" Oct 31 19:19:26.752268 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.752244 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 19:19:26.752349 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.752270 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 19:19:26.752349 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.752280 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 19:19:26.752349 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.752286 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bbst6\"" Oct 31 19:19:26.752594 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.752579 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4hvl4\"" Oct 31 19:19:26.752661 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.752599 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-j7nlh\"" Oct 31 19:19:26.752661 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.752628 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cs8fb\"" Oct 31 19:19:26.752661 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.752614 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mrxc5\"" Oct 31 19:19:26.752661 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.752632 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 19:19:26.752661 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.752646 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rs77w\"" Oct 31 19:19:26.752661 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.752662 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q7xwb\"" Oct 31 19:19:26.752832 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.752671 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 19:19:26.752832 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.752681 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vq9kc\"" Oct 31 19:19:26.754457 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.754444 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 19:19:26.758329 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.758307 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 19:19:26.758412 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.758307 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 19:19:26.758467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.758453 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 19:19:26.804590 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.804567 4910 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:19:26.815094 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.815068 4910 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 19:19:26.818044 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.818022 4910 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 19:19:26.818147 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:26.818065 4910 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-116.us-east-2.compute.internal" Oct 31 19:19:26.838418 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.838390 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e729724-d310-4884-a2d0-54af5308124c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:26.838534 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.838422 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-run-systemd\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.838534 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.838442 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2552d079-24ff-4651-b77e-155af2fe48e9-iptables-alerter-script\") pod \"iptables-alerter-qn24x\" (UID: \"2552d079-24ff-4651-b77e-155af2fe48e9\") " pod="openshift-network-operator/iptables-alerter-qn24x" Oct 31 19:19:26.838534 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.838465 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:19:26.838682 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.838548 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-sys-fs\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:19:26.838682 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.838590 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w48sj\" (UniqueName: \"kubernetes.io/projected/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-kube-api-access-w48sj\") pod \"cluster-samples-operator-6ddb6c8b94-9q7w4\" (UID: \"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:26.838682 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.838618 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b42xc\" (UniqueName: \"kubernetes.io/projected/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-kube-api-access-b42xc\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.838682 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.838658 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb1a9bea-399b-4281-9ada-a232790cc713-trusted-ca\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:26.838899 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.838685 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/84f5548f-7d70-4391-89c9-3bd3c1784717-serviceca\") pod \"node-ca-8m482\" (UID: \"84f5548f-7d70-4391-89c9-3bd3c1784717\") " pod="openshift-image-registry/node-ca-8m482" Oct 31 19:19:26.838899 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.838710 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpcv\" (UniqueName: \"kubernetes.io/projected/4cd15118-0a59-4122-bdf1-56710c463262-kube-api-access-nxpcv\") pod \"cluster-monitoring-operator-884576c57-d4kds\" (UID: \"4cd15118-0a59-4122-bdf1-56710c463262\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" Oct 31 19:19:26.838899 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.838740 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-var-lib-openvswitch\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.838899 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.838763 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-sysconfig\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.838899 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.838813 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92aaddbc-6268-443c-b3bd-9f74ab6b9a4b-cert\") pod \"ingress-canary-q8j58\" (UID: \"92aaddbc-6268-443c-b3bd-9f74ab6b9a4b\") " pod="openshift-ingress-canary/ingress-canary-q8j58" Oct 31 19:19:26.838899 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.838837 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7djgh\" (UniqueName: \"kubernetes.io/projected/23a10416-51b1-415f-b2fc-892b4d662b1f-kube-api-access-7djgh\") pod \"network-check-target-5mkxz\" (UID: \"23a10416-51b1-415f-b2fc-892b4d662b1f\") " pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:19:26.838899 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.838861 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5418e7b8-1471-4aa5-ad70-5d60b87c572a-config-volume\") pod \"dns-default-5nm2g\" (UID: \"5418e7b8-1471-4aa5-ad70-5d60b87c572a\") " pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:26.839323 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.838894 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5418e7b8-1471-4aa5-ad70-5d60b87c572a-tmp-dir\") pod \"dns-default-5nm2g\" (UID: \"5418e7b8-1471-4aa5-ad70-5d60b87c572a\") " pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:26.839323 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.838902 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2552d079-24ff-4651-b77e-155af2fe48e9-iptables-alerter-script\") pod \"iptables-alerter-qn24x\" (UID: \"2552d079-24ff-4651-b77e-155af2fe48e9\") " pod="openshift-network-operator/iptables-alerter-qn24x" Oct 31 19:19:26.839323 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.838955 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e729724-d310-4884-a2d0-54af5308124c-data-volume\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:26.839323 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839024 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81405623-de29-4fc6-bccf-447967669d89-ovnkube-script-lib\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.839323 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839049 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-etc-kubernetes\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.839323 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839075 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5418e7b8-1471-4aa5-ad70-5d60b87c572a-tmp-dir\") pod \"dns-default-5nm2g\" (UID: \"5418e7b8-1471-4aa5-ad70-5d60b87c572a\") " pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:26.839323 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839126 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5418e7b8-1471-4aa5-ad70-5d60b87c572a-config-volume\") pod \"dns-default-5nm2g\" (UID: \"5418e7b8-1471-4aa5-ad70-5d60b87c572a\") " pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:26.839323 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839121 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-run\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.839323 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839160 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb1a9bea-399b-4281-9ada-a232790cc713-trusted-ca\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:26.839323 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839195 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e729724-d310-4884-a2d0-54af5308124c-data-volume\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:26.839323 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839198 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e729724-d310-4884-a2d0-54af5308124c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:26.839323 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839236 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zkn5\" (UniqueName: \"kubernetes.io/projected/ffb7495b-ef69-455c-99aa-22389d8c6d6e-kube-api-access-9zkn5\") pod \"volume-data-source-validator-58c9f5f46d-xgrs8\" (UID: \"ffb7495b-ef69-455c-99aa-22389d8c6d6e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgrs8" Oct 31 19:19:26.839323 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839251 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92aaddbc-6268-443c-b3bd-9f74ab6b9a4b-cert\") pod \"ingress-canary-q8j58\" (UID: \"92aaddbc-6268-443c-b3bd-9f74ab6b9a4b\") " pod="openshift-ingress-canary/ingress-canary-q8j58" Oct 31 19:19:26.839940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839368 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81405623-de29-4fc6-bccf-447967669d89-ovnkube-script-lib\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.839940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839375 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5trh\" (UniqueName: \"kubernetes.io/projected/3a7f7d6d-958c-4973-9289-97f45526a431-kube-api-access-q5trh\") pod \"kas-connection-checker-685895c7c9-4mdb2\" (UID: \"3a7f7d6d-958c-4973-9289-97f45526a431\") " pod="kube-system/kas-connection-checker-685895c7c9-4mdb2" Oct 31 19:19:26.839940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839405 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1d0b685c-252c-459c-9dba-d19c1bc6e5d0-agent-certs\") pod \"konnectivity-agent-wxdtf\" (UID: \"1d0b685c-252c-459c-9dba-d19c1bc6e5d0\") " pod="kube-system/konnectivity-agent-wxdtf" Oct 31 19:19:26.839940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839432 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/599fc0e5-05d8-46b0-90b3-60818a371161-snapshots\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:26.839940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839443 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/84f5548f-7d70-4391-89c9-3bd3c1784717-serviceca\") pod \"node-ca-8m482\" (UID: \"84f5548f-7d70-4391-89c9-3bd3c1784717\") " pod="openshift-image-registry/node-ca-8m482" Oct 31 19:19:26.839940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839504 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/599fc0e5-05d8-46b0-90b3-60818a371161-snapshots\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:26.839940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839540 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4wnr\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-kube-api-access-t4wnr\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:26.839940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839583 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzwvn\" (UniqueName: \"kubernetes.io/projected/5418e7b8-1471-4aa5-ad70-5d60b87c572a-kube-api-access-vzwvn\") pod \"dns-default-5nm2g\" (UID: \"5418e7b8-1471-4aa5-ad70-5d60b87c572a\") " pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:26.839940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839612 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zsbj\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-kube-api-access-4zsbj\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:26.839940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839638 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81405623-de29-4fc6-bccf-447967669d89-env-overrides\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.839940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839680 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-kubelet\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.839940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839704 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-cni-bin\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.839940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839730 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47cff5a0-505b-45e9-9788-cbe4b557917e-system-cni-dir\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:19:26.839940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839680 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1d0b685c-252c-459c-9dba-d19c1bc6e5d0-agent-certs\") pod \"konnectivity-agent-wxdtf\" (UID: \"1d0b685c-252c-459c-9dba-d19c1bc6e5d0\") " pod="kube-system/konnectivity-agent-wxdtf" Oct 31 19:19:26.839940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839762 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-hostroot\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.839940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839787 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-systemd\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.839940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839837 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81405623-de29-4fc6-bccf-447967669d89-env-overrides\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.839940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839855 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76e07820-d888-4138-a41a-59ff4565accd-trusted-ca\") pod \"console-operator-7cd56b8dbc-6vpwc\" (UID: \"76e07820-d888-4138-a41a-59ff4565accd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:26.839940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839872 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-var-lib-cni-bin\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.839940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839887 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.839940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.839936 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4cd15118-0a59-4122-bdf1-56710c463262-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-d4kds\" (UID: \"4cd15118-0a59-4122-bdf1-56710c463262\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" Oct 31 19:19:26.840659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840062 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-cnibin\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.840659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840108 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9255a1d5-2514-441e-b942-6c39843e38f8-dbus\") pod \"global-pull-secret-syncer-lhtkb\" (UID: \"9255a1d5-2514-441e-b942-6c39843e38f8\") " pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:19:26.840659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840175 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4cd15118-0a59-4122-bdf1-56710c463262-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-d4kds\" (UID: \"4cd15118-0a59-4122-bdf1-56710c463262\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" Oct 31 19:19:26.840659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840173 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-nginx-conf\") pod \"networking-console-plugin-86c478bb88-28mzh\" (UID: \"9cf7d7f5-3936-45b3-ba2c-597cf99cfb89\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-28mzh" Oct 31 19:19:26.840659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840194 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/76e07820-d888-4138-a41a-59ff4565accd-trusted-ca\") pod \"console-operator-7cd56b8dbc-6vpwc\" (UID: \"76e07820-d888-4138-a41a-59ff4565accd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:26.840659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840213 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-stats-auth\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:26.840659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840231 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:26.840659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840255 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jzkh\" (UniqueName: \"kubernetes.io/projected/47cff5a0-505b-45e9-9788-cbe4b557917e-kube-api-access-8jzkh\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:19:26.840659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840280 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-run-netns\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.840659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840336 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-nginx-conf\") pod \"networking-console-plugin-86c478bb88-28mzh\" (UID: \"9cf7d7f5-3936-45b3-ba2c-597cf99cfb89\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-28mzh" Oct 31 19:19:26.840659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840399 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-modprobe-d\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.840659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840439 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-stats-auth\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:26.840659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840448 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb1a9bea-399b-4281-9ada-a232790cc713-image-registry-private-configuration\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:26.840659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840581 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msgz5\" (UniqueName: \"kubernetes.io/projected/84f5548f-7d70-4391-89c9-3bd3c1784717-kube-api-access-msgz5\") pod \"node-ca-8m482\" (UID: \"84f5548f-7d70-4391-89c9-3bd3c1784717\") " pod="openshift-image-registry/node-ca-8m482" Oct 31 19:19:26.840659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840602 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qtrj\" (UniqueName: \"kubernetes.io/projected/4ce261ba-08ff-418a-b3b4-b7e25e701bd8-kube-api-access-4qtrj\") pod \"service-ca-operator-544d749c8c-kts6n\" (UID: \"4ce261ba-08ff-418a-b3b4-b7e25e701bd8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n" Oct 31 19:19:26.840659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840609 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb1a9bea-399b-4281-9ada-a232790cc713-image-registry-private-configuration\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:26.840659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840620 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2552d079-24ff-4651-b77e-155af2fe48e9-host-slash\") pod \"iptables-alerter-qn24x\" (UID: \"2552d079-24ff-4651-b77e-155af2fe48e9\") " pod="openshift-network-operator/iptables-alerter-qn24x" Oct 31 19:19:26.840659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840655 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5708be73-8282-4caa-bce3-46d4b708860e-multus-daemon-config\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.840659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840673 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-tuned\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.841437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840690 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-d4kds\" (UID: \"4cd15118-0a59-4122-bdf1-56710c463262\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" Oct 31 19:19:26.841437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840706 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-systemd-units\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.841437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840721 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/599fc0e5-05d8-46b0-90b3-60818a371161-tmp\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:26.841437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840747 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a063ed7-8ce0-414e-8bbf-7733c464b507-trusted-ca\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:26.841437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840770 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-tuned\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.841437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840878 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/599fc0e5-05d8-46b0-90b3-60818a371161-tmp\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:26.841437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840924 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cj8g\" (UniqueName: \"kubernetes.io/projected/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-kube-api-access-4cj8g\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:26.841437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.840965 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5708be73-8282-4caa-bce3-46d4b708860e-multus-daemon-config\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.841437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841053 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn2tq\" (UniqueName: \"kubernetes.io/projected/4e729724-d310-4884-a2d0-54af5308124c-kube-api-access-fn2tq\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:26.841437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841089 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81405623-de29-4fc6-bccf-447967669d89-ovnkube-config\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.841437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841116 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7whr\" (UniqueName: \"kubernetes.io/projected/81405623-de29-4fc6-bccf-447967669d89-kube-api-access-l7whr\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.841437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841142 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-run-netns\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.841437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841169 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-etc-openvswitch\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.841437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841205 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5708be73-8282-4caa-bce3-46d4b708860e-cni-binary-copy\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.841437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841244 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a063ed7-8ce0-414e-8bbf-7733c464b507-trusted-ca\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:26.841437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841277 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9q7w4\" (UID: \"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:26.841437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841301 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81405623-de29-4fc6-bccf-447967669d89-ovnkube-config\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.841437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841324 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9255a1d5-2514-441e-b942-6c39843e38f8-original-pull-secret\") pod \"global-pull-secret-syncer-lhtkb\" (UID: \"9255a1d5-2514-441e-b942-6c39843e38f8\") " pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:19:26.841437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841368 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5708be73-8282-4caa-bce3-46d4b708860e-cni-binary-copy\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.841437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841374 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g49vm\" (UniqueName: \"kubernetes.io/projected/3e84edf8-6e35-489b-b19b-17a5179e666e-kube-api-access-g49vm\") pod \"network-check-source-6f78d59f99-l84pf\" (UID: \"3e84edf8-6e35-489b-b19b-17a5179e666e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l84pf" Oct 31 19:19:26.841437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841396 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1d0b685c-252c-459c-9dba-d19c1bc6e5d0-konnectivity-ca\") pod \"konnectivity-agent-wxdtf\" (UID: \"1d0b685c-252c-459c-9dba-d19c1bc6e5d0\") " pod="kube-system/konnectivity-agent-wxdtf" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841498 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76e07820-d888-4138-a41a-59ff4565accd-config\") pod \"console-operator-7cd56b8dbc-6vpwc\" (UID: \"76e07820-d888-4138-a41a-59ff4565accd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841518 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqspp\" (UniqueName: \"kubernetes.io/projected/76e07820-d888-4138-a41a-59ff4565accd-kube-api-access-dqspp\") pod \"console-operator-7cd56b8dbc-6vpwc\" (UID: \"76e07820-d888-4138-a41a-59ff4565accd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841537 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mk7v\" (UniqueName: \"kubernetes.io/projected/211efbea-29d0-4b2f-8c9e-6f3badcf77e5-kube-api-access-9mk7v\") pod \"kas-connection-checker-685895c7c9-vszkf\" (UID: \"211efbea-29d0-4b2f-8c9e-6f3badcf77e5\") " pod="kube-system/kas-connection-checker-685895c7c9-vszkf" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841549 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1d0b685c-252c-459c-9dba-d19c1bc6e5d0-konnectivity-ca\") pod \"konnectivity-agent-wxdtf\" (UID: \"1d0b685c-252c-459c-9dba-d19c1bc6e5d0\") " pod="kube-system/konnectivity-agent-wxdtf" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841558 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb1a9bea-399b-4281-9ada-a232790cc713-registry-certificates\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841567 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9255a1d5-2514-441e-b942-6c39843e38f8-original-pull-secret\") pod \"global-pull-secret-syncer-lhtkb\" (UID: \"9255a1d5-2514-441e-b942-6c39843e38f8\") " pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841594 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-var-lib-kubelet\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841620 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9255a1d5-2514-441e-b942-6c39843e38f8-kubelet-config\") pod \"global-pull-secret-syncer-lhtkb\" (UID: \"9255a1d5-2514-441e-b942-6c39843e38f8\") " pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841648 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-certificates\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841658 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76e07820-d888-4138-a41a-59ff4565accd-config\") pod \"console-operator-7cd56b8dbc-6vpwc\" (UID: \"76e07820-d888-4138-a41a-59ff4565accd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841674 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-run-openvswitch\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841700 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb1a9bea-399b-4281-9ada-a232790cc713-ca-trust-extracted\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841723 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81405623-de29-4fc6-bccf-447967669d89-ovn-node-metrics-cert\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841766 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47cff5a0-505b-45e9-9788-cbe4b557917e-cnibin\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841797 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/599fc0e5-05d8-46b0-90b3-60818a371161-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841809 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb1a9bea-399b-4281-9ada-a232790cc713-ca-trust-extracted\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841846 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-sys\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841879 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-58hlm\" (UID: \"ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841952 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-var-lib-kubelet\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.841986 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mjmg\" (UniqueName: \"kubernetes.io/projected/5708be73-8282-4caa-bce3-46d4b708860e-kube-api-access-9mjmg\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842016 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5418e7b8-1471-4aa5-ad70-5d60b87c572a-metrics-tls\") pod \"dns-default-5nm2g\" (UID: \"5418e7b8-1471-4aa5-ad70-5d60b87c572a\") " pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842046 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7fw4\" (UniqueName: \"kubernetes.io/projected/7bce545d-6640-49e8-848c-aa9f8db19902-kube-api-access-l7fw4\") pod \"migrator-66df5d496b-4fsp7\" (UID: \"7bce545d-6640-49e8-848c-aa9f8db19902\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4fsp7" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842074 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/599fc0e5-05d8-46b0-90b3-60818a371161-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842075 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnwvq\" (UniqueName: \"kubernetes.io/projected/52432f95-9eb4-4fcd-b795-e475f3399c62-kube-api-access-pnwvq\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842107 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-multus-cni-dir\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842125 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/599fc0e5-05d8-46b0-90b3-60818a371161-serving-cert\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842126 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81405623-de29-4fc6-bccf-447967669d89-ovn-node-metrics-cert\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842142 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-bound-sa-token\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842157 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47cff5a0-505b-45e9-9788-cbe4b557917e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842268 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/599fc0e5-05d8-46b0-90b3-60818a371161-serving-cert\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842273 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-certificates\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842299 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-system-cni-dir\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842316 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-tmp\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842353 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a063ed7-8ce0-414e-8bbf-7733c464b507-ca-trust-extracted\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842369 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ce261ba-08ff-418a-b3b4-b7e25e701bd8-serving-cert\") pod \"service-ca-operator-544d749c8c-kts6n\" (UID: \"4ce261ba-08ff-418a-b3b4-b7e25e701bd8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842385 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47cff5a0-505b-45e9-9788-cbe4b557917e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842401 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-var-lib-cni-multus\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842433 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-58hlm\" (UID: \"ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842449 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47cff5a0-505b-45e9-9788-cbe4b557917e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842453 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-default-certificate\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842479 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb1a9bea-399b-4281-9ada-a232790cc713-registry-certificates\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842490 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842511 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb1a9bea-399b-4281-9ada-a232790cc713-installation-pull-secrets\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842518 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-tmp\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842542 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-socket-dir\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:19:26.842579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842613 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a063ed7-8ce0-414e-8bbf-7733c464b507-ca-trust-extracted\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842644 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-default-certificate\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842648 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-device-dir\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842675 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-etc-selinux\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842709 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ce261ba-08ff-418a-b3b4-b7e25e701bd8-serving-cert\") pod \"service-ca-operator-544d749c8c-kts6n\" (UID: \"4ce261ba-08ff-418a-b3b4-b7e25e701bd8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842726 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb1a9bea-399b-4281-9ada-a232790cc713-installation-pull-secrets\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842739 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47cff5a0-505b-45e9-9788-cbe4b557917e-cni-binary-copy\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842775 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q2cm\" (UniqueName: \"kubernetes.io/projected/88a1fa4b-1191-434a-8e7c-783163ece24d-kube-api-access-9q2cm\") pod \"network-metrics-daemon-dfbrp\" (UID: \"88a1fa4b-1191-434a-8e7c-783163ece24d\") " pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842880 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c13795-a1de-4761-b3fc-feef76ac730d-tmp-dir\") pod \"node-resolver-85mq2\" (UID: \"28c13795-a1de-4761-b3fc-feef76ac730d\") " pod="openshift-dns/node-resolver-85mq2" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842944 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a063ed7-8ce0-414e-8bbf-7733c464b507-image-registry-private-configuration\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842969 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c13795-a1de-4761-b3fc-feef76ac730d-tmp-dir\") pod \"node-resolver-85mq2\" (UID: \"28c13795-a1de-4761-b3fc-feef76ac730d\") " pod="openshift-dns/node-resolver-85mq2" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.842976 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a063ed7-8ce0-414e-8bbf-7733c464b507-installation-pull-secrets\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843023 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47cff5a0-505b-45e9-9788-cbe4b557917e-cni-binary-copy\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843072 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76e07820-d888-4138-a41a-59ff4565accd-serving-cert\") pod \"console-operator-7cd56b8dbc-6vpwc\" (UID: \"76e07820-d888-4138-a41a-59ff4565accd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843105 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a063ed7-8ce0-414e-8bbf-7733c464b507-image-registry-private-configuration\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843151 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-run-ovn\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843203 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a063ed7-8ce0-414e-8bbf-7733c464b507-installation-pull-secrets\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843232 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76e07820-d888-4138-a41a-59ff4565accd-serving-cert\") pod \"console-operator-7cd56b8dbc-6vpwc\" (UID: \"76e07820-d888-4138-a41a-59ff4565accd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843256 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-cni-netd\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843283 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47cff5a0-505b-45e9-9788-cbe4b557917e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843299 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-9q7w4\" (UID: \"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843340 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-node-log\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843387 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-run-k8s-cni-cncf-io\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843417 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4lhh\" (UniqueName: \"kubernetes.io/projected/599fc0e5-05d8-46b0-90b3-60818a371161-kube-api-access-b4lhh\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843419 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-9q7w4\" (UID: \"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843440 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-bound-sa-token\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843452 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47cff5a0-505b-45e9-9788-cbe4b557917e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843465 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e729724-d310-4884-a2d0-54af5308124c-crio-socket\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843508 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ce261ba-08ff-418a-b3b4-b7e25e701bd8-config\") pod \"service-ca-operator-544d749c8c-kts6n\" (UID: \"4ce261ba-08ff-418a-b3b4-b7e25e701bd8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843533 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843551 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-multus-socket-dir-parent\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843579 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-sysctl-conf\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843616 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-host\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843644 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp54m\" (UniqueName: \"kubernetes.io/projected/ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f-kube-api-access-gp54m\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-58hlm\" (UID: \"ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843671 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxnqt\" (UniqueName: \"kubernetes.io/projected/92aaddbc-6268-443c-b3bd-9f74ab6b9a4b-kube-api-access-rxnqt\") pod \"ingress-canary-q8j58\" (UID: \"92aaddbc-6268-443c-b3bd-9f74ab6b9a4b\") " pod="openshift-ingress-canary/ingress-canary-q8j58" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843707 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ce261ba-08ff-418a-b3b4-b7e25e701bd8-config\") pod \"service-ca-operator-544d749c8c-kts6n\" (UID: \"4ce261ba-08ff-418a-b3b4-b7e25e701bd8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843762 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843868 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd4t7\" (UniqueName: \"kubernetes.io/projected/28c13795-a1de-4761-b3fc-feef76ac730d-kube-api-access-bd4t7\") pod \"node-resolver-85mq2\" (UID: \"28c13795-a1de-4761-b3fc-feef76ac730d\") " pod="openshift-dns/node-resolver-85mq2" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843898 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-log-socket\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.843934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843945 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-run-ovn-kubernetes\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.845234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843962 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-multus-conf-dir\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.845234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.843990 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-kubernetes\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.845234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.844034 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-run-multus-certs\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.845234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.844066 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88a1fa4b-1191-434a-8e7c-783163ece24d-metrics-certs\") pod \"network-metrics-daemon-dfbrp\" (UID: \"88a1fa4b-1191-434a-8e7c-783163ece24d\") " pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:19:26.845234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.844091 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-lib-modules\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.845234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.844110 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-28mzh\" (UID: \"9cf7d7f5-3936-45b3-ba2c-597cf99cfb89\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-28mzh" Oct 31 19:19:26.845234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.844148 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/28c13795-a1de-4761-b3fc-feef76ac730d-hosts-file\") pod \"node-resolver-85mq2\" (UID: \"28c13795-a1de-4761-b3fc-feef76ac730d\") " pod="openshift-dns/node-resolver-85mq2" Oct 31 19:19:26.845234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.844208 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5mrm\" (UniqueName: \"kubernetes.io/projected/2552d079-24ff-4651-b77e-155af2fe48e9-kube-api-access-t5mrm\") pod \"iptables-alerter-qn24x\" (UID: \"2552d079-24ff-4651-b77e-155af2fe48e9\") " pod="openshift-network-operator/iptables-alerter-qn24x" Oct 31 19:19:26.845234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.844243 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-registration-dir\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:19:26.845234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.844282 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/599fc0e5-05d8-46b0-90b3-60818a371161-service-ca-bundle\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:26.845234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.844301 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-58hlm\" (UID: \"ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm" Oct 31 19:19:26.845234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.844328 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/84f5548f-7d70-4391-89c9-3bd3c1784717-host\") pod \"node-ca-8m482\" (UID: \"84f5548f-7d70-4391-89c9-3bd3c1784717\") " pod="openshift-image-registry/node-ca-8m482" Oct 31 19:19:26.845234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.844416 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-slash\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.845234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.844435 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6wxr\" (UniqueName: \"kubernetes.io/projected/1cc2220b-f30e-4879-b2cb-b2dd8e53b9f0-kube-api-access-z6wxr\") pod \"kas-connection-checker-685895c7c9-xmfjb\" (UID: \"1cc2220b-f30e-4879-b2cb-b2dd8e53b9f0\") " pod="kube-system/kas-connection-checker-685895c7c9-xmfjb" Oct 31 19:19:26.845234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.844457 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/599fc0e5-05d8-46b0-90b3-60818a371161-service-ca-bundle\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:26.845234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.844477 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-sysctl-d\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.845234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.844506 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-58hlm\" (UID: \"ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm" Oct 31 19:19:26.858798 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.858776 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w48sj\" (UniqueName: \"kubernetes.io/projected/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-kube-api-access-w48sj\") pod \"cluster-samples-operator-6ddb6c8b94-9q7w4\" (UID: \"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:26.864223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.864203 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn2tq\" (UniqueName: \"kubernetes.io/projected/4e729724-d310-4884-a2d0-54af5308124c-kube-api-access-fn2tq\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:26.865272 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.865256 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnwvq\" (UniqueName: \"kubernetes.io/projected/52432f95-9eb4-4fcd-b795-e475f3399c62-kube-api-access-pnwvq\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:19:26.865684 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.865670 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5trh\" (UniqueName: \"kubernetes.io/projected/3a7f7d6d-958c-4973-9289-97f45526a431-kube-api-access-q5trh\") pod \"kas-connection-checker-685895c7c9-4mdb2\" (UID: \"3a7f7d6d-958c-4973-9289-97f45526a431\") " pod="kube-system/kas-connection-checker-685895c7c9-4mdb2" Oct 31 19:19:26.877105 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.877084 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-bound-sa-token\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:26.877174 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.877125 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp54m\" (UniqueName: \"kubernetes.io/projected/ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f-kube-api-access-gp54m\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-58hlm\" (UID: \"ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm" Oct 31 19:19:26.879858 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.879836 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7djgh\" (UniqueName: \"kubernetes.io/projected/23a10416-51b1-415f-b2fc-892b4d662b1f-kube-api-access-7djgh\") pod \"network-check-target-5mkxz\" (UID: \"23a10416-51b1-415f-b2fc-892b4d662b1f\") " pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:19:26.880051 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.880037 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqspp\" (UniqueName: \"kubernetes.io/projected/76e07820-d888-4138-a41a-59ff4565accd-kube-api-access-dqspp\") pod \"console-operator-7cd56b8dbc-6vpwc\" (UID: \"76e07820-d888-4138-a41a-59ff4565accd\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:26.880285 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.880268 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4lhh\" (UniqueName: \"kubernetes.io/projected/599fc0e5-05d8-46b0-90b3-60818a371161-kube-api-access-b4lhh\") pod \"insights-operator-7c4986d449-mtmp6\" (UID: \"599fc0e5-05d8-46b0-90b3-60818a371161\") " pod="openshift-insights/insights-operator-7c4986d449-mtmp6" Oct 31 19:19:26.880561 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.880546 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7fw4\" (UniqueName: \"kubernetes.io/projected/7bce545d-6640-49e8-848c-aa9f8db19902-kube-api-access-l7fw4\") pod \"migrator-66df5d496b-4fsp7\" (UID: \"7bce545d-6640-49e8-848c-aa9f8db19902\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4fsp7" Oct 31 19:19:26.880665 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.880648 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxpcv\" (UniqueName: \"kubernetes.io/projected/4cd15118-0a59-4122-bdf1-56710c463262-kube-api-access-nxpcv\") pod \"cluster-monitoring-operator-884576c57-d4kds\" (UID: \"4cd15118-0a59-4122-bdf1-56710c463262\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" Oct 31 19:19:26.881101 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.881086 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jzkh\" (UniqueName: \"kubernetes.io/projected/47cff5a0-505b-45e9-9788-cbe4b557917e-kube-api-access-8jzkh\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:19:26.881259 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.881244 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zsbj\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-kube-api-access-4zsbj\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:26.882309 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.882294 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msgz5\" (UniqueName: \"kubernetes.io/projected/84f5548f-7d70-4391-89c9-3bd3c1784717-kube-api-access-msgz5\") pod \"node-ca-8m482\" (UID: \"84f5548f-7d70-4391-89c9-3bd3c1784717\") " pod="openshift-image-registry/node-ca-8m482" Oct 31 19:19:26.890182 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.890168 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-bound-sa-token\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:26.890182 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.890176 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd4t7\" (UniqueName: \"kubernetes.io/projected/28c13795-a1de-4761-b3fc-feef76ac730d-kube-api-access-bd4t7\") pod \"node-resolver-85mq2\" (UID: \"28c13795-a1de-4761-b3fc-feef76ac730d\") " pod="openshift-dns/node-resolver-85mq2" Oct 31 19:19:26.890505 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.890487 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zkn5\" (UniqueName: \"kubernetes.io/projected/ffb7495b-ef69-455c-99aa-22389d8c6d6e-kube-api-access-9zkn5\") pod \"volume-data-source-validator-58c9f5f46d-xgrs8\" (UID: \"ffb7495b-ef69-455c-99aa-22389d8c6d6e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-xgrs8" Oct 31 19:19:26.892441 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.892426 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7whr\" (UniqueName: \"kubernetes.io/projected/81405623-de29-4fc6-bccf-447967669d89-kube-api-access-l7whr\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.906641 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.906549 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mk7v\" (UniqueName: \"kubernetes.io/projected/211efbea-29d0-4b2f-8c9e-6f3badcf77e5-kube-api-access-9mk7v\") pod \"kas-connection-checker-685895c7c9-vszkf\" (UID: \"211efbea-29d0-4b2f-8c9e-6f3badcf77e5\") " pod="kube-system/kas-connection-checker-685895c7c9-vszkf" Oct 31 19:19:26.906641 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.906572 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q2cm\" (UniqueName: \"kubernetes.io/projected/88a1fa4b-1191-434a-8e7c-783163ece24d-kube-api-access-9q2cm\") pod \"network-metrics-daemon-dfbrp\" (UID: \"88a1fa4b-1191-434a-8e7c-783163ece24d\") " pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:19:26.906641 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.906636 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mjmg\" (UniqueName: \"kubernetes.io/projected/5708be73-8282-4caa-bce3-46d4b708860e-kube-api-access-9mjmg\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.906960 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.906706 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5mrm\" (UniqueName: \"kubernetes.io/projected/2552d079-24ff-4651-b77e-155af2fe48e9-kube-api-access-t5mrm\") pod \"iptables-alerter-qn24x\" (UID: \"2552d079-24ff-4651-b77e-155af2fe48e9\") " pod="openshift-network-operator/iptables-alerter-qn24x" Oct 31 19:19:26.906960 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.906724 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6wxr\" (UniqueName: \"kubernetes.io/projected/1cc2220b-f30e-4879-b2cb-b2dd8e53b9f0-kube-api-access-z6wxr\") pod \"kas-connection-checker-685895c7c9-xmfjb\" (UID: \"1cc2220b-f30e-4879-b2cb-b2dd8e53b9f0\") " pod="kube-system/kas-connection-checker-685895c7c9-xmfjb" Oct 31 19:19:26.906960 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.906779 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b42xc\" (UniqueName: \"kubernetes.io/projected/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-kube-api-access-b42xc\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.906960 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.906819 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxnqt\" (UniqueName: \"kubernetes.io/projected/92aaddbc-6268-443c-b3bd-9f74ab6b9a4b-kube-api-access-rxnqt\") pod \"ingress-canary-q8j58\" (UID: \"92aaddbc-6268-443c-b3bd-9f74ab6b9a4b\") " pod="openshift-ingress-canary/ingress-canary-q8j58" Oct 31 19:19:26.907177 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.907050 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzwvn\" (UniqueName: \"kubernetes.io/projected/5418e7b8-1471-4aa5-ad70-5d60b87c572a-kube-api-access-vzwvn\") pod \"dns-default-5nm2g\" (UID: \"5418e7b8-1471-4aa5-ad70-5d60b87c572a\") " pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:26.907177 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.907105 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qtrj\" (UniqueName: \"kubernetes.io/projected/4ce261ba-08ff-418a-b3b4-b7e25e701bd8-kube-api-access-4qtrj\") pod \"service-ca-operator-544d749c8c-kts6n\" (UID: \"4ce261ba-08ff-418a-b3b4-b7e25e701bd8\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n" Oct 31 19:19:26.907539 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.907514 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g49vm\" (UniqueName: \"kubernetes.io/projected/3e84edf8-6e35-489b-b19b-17a5179e666e-kube-api-access-g49vm\") pod \"network-check-source-6f78d59f99-l84pf\" (UID: \"3e84edf8-6e35-489b-b19b-17a5179e666e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-l84pf" Oct 31 19:19:26.907603 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.907517 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cj8g\" (UniqueName: \"kubernetes.io/projected/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-kube-api-access-4cj8g\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:26.907603 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.907552 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4wnr\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-kube-api-access-t4wnr\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:26.945740 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.945715 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-multus-cni-dir\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.945740 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.945743 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-system-cni-dir\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.945935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.945760 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47cff5a0-505b-45e9-9788-cbe4b557917e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:19:26.945935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.945776 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-var-lib-cni-multus\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.945935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.945793 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:26.945935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.945816 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-socket-dir\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:19:26.945935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.945838 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-device-dir\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:19:26.945935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.945844 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-system-cni-dir\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.945935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.945856 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-multus-cni-dir\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.945935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.945866 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47cff5a0-505b-45e9-9788-cbe4b557917e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:19:26.945935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.945860 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-var-lib-cni-multus\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.945935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.945903 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-socket-dir\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:19:26.945935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.945930 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-device-dir\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:19:26.946292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.945957 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-etc-selinux\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:19:26.946292 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:26.945980 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle podName:70c870bc-c9e6-4d7e-bc99-ba556ce31cde nodeName:}" failed. No retries permitted until 2026-10-31 19:19:27.445959651 +0000 UTC m=+1.808780019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle") pod "router-default-8c75d8d7f-tzzdl" (UID: "70c870bc-c9e6-4d7e-bc99-ba556ce31cde") : configmap references non-existent config key: service-ca.crt Oct 31 19:19:26.946292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946018 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-run-ovn\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.946292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946024 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-etc-selinux\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:19:26.946292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946053 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-cni-netd\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.946292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946059 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-run-ovn\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.946292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946083 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-node-log\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.946292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946098 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-cni-netd\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.946292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946106 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-node-log\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.946292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946108 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-run-k8s-cni-cncf-io\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.946292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946138 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e729724-d310-4884-a2d0-54af5308124c-crio-socket\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:26.946292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946147 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-run-k8s-cni-cncf-io\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.946292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946158 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:26.946292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946180 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-multus-socket-dir-parent\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.946292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946214 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e729724-d310-4884-a2d0-54af5308124c-crio-socket\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:26.946292 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:26.946241 4910 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 19:19:26.946292 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:26.946253 4910 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8ccd4bc6-xlml9: secret "image-registry-tls" not found Oct 31 19:19:26.946292 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:26.946292 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls podName:3a063ed7-8ce0-414e-8bbf-7733c464b507 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:27.446283366 +0000 UTC m=+1.809103752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls") pod "image-registry-5c8ccd4bc6-xlml9" (UID: "3a063ed7-8ce0-414e-8bbf-7733c464b507") : secret "image-registry-tls" not found Oct 31 19:19:26.946292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946293 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-multus-socket-dir-parent\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.946857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946582 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-sysctl-conf\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.946857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946602 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-host\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.946857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946619 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:26.946857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946633 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-log-socket\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.946857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946647 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-run-ovn-kubernetes\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.946857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946667 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-multus-conf-dir\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.946857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946674 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-host\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.946857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946700 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-log-socket\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.946857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946702 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-multus-conf-dir\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.946857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946714 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-kubernetes\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.946857 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:26.946739 4910 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 19:19:26.946857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946738 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-run-ovn-kubernetes\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.946857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946747 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-run-multus-certs\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.946857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946774 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88a1fa4b-1191-434a-8e7c-783163ece24d-metrics-certs\") pod \"network-metrics-daemon-dfbrp\" (UID: \"88a1fa4b-1191-434a-8e7c-783163ece24d\") " pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:19:26.946857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946780 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-run-multus-certs\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.946857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946742 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-sysctl-conf\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.946857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946799 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-lib-modules\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.946857 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:26.946750 4910 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f878896cc-hzn9v: secret "image-registry-tls" not found Oct 31 19:19:26.946857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946752 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-kubernetes\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.946857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946825 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-28mzh\" (UID: \"9cf7d7f5-3936-45b3-ba2c-597cf99cfb89\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-28mzh" Oct 31 19:19:26.946857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946847 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/28c13795-a1de-4761-b3fc-feef76ac730d-hosts-file\") pod \"node-resolver-85mq2\" (UID: \"28c13795-a1de-4761-b3fc-feef76ac730d\") " pod="openshift-dns/node-resolver-85mq2" Oct 31 19:19:26.946857 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:26.946866 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls podName:eb1a9bea-399b-4281-9ada-a232790cc713 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:27.446854204 +0000 UTC m=+1.809674553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls") pod "image-registry-6f878896cc-hzn9v" (UID: "eb1a9bea-399b-4281-9ada-a232790cc713") : secret "image-registry-tls" not found Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946880 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/28c13795-a1de-4761-b3fc-feef76ac730d-hosts-file\") pod \"node-resolver-85mq2\" (UID: \"28c13795-a1de-4761-b3fc-feef76ac730d\") " pod="openshift-dns/node-resolver-85mq2" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946887 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-registration-dir\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946896 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-lib-modules\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946929 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/84f5548f-7d70-4391-89c9-3bd3c1784717-host\") pod \"node-ca-8m482\" (UID: \"84f5548f-7d70-4391-89c9-3bd3c1784717\") " pod="openshift-image-registry/node-ca-8m482" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:26.946946 4910 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946950 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-registration-dir\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946946 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-slash\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946968 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-slash\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.946978 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/84f5548f-7d70-4391-89c9-3bd3c1784717-host\") pod \"node-ca-8m482\" (UID: \"84f5548f-7d70-4391-89c9-3bd3c1784717\") " pod="openshift-image-registry/node-ca-8m482" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:26.946986 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert podName:9cf7d7f5-3936-45b3-ba2c-597cf99cfb89 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:27.446973741 +0000 UTC m=+1.809794124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-28mzh" (UID: "9cf7d7f5-3936-45b3-ba2c-597cf99cfb89") : secret "networking-console-plugin-cert" not found Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947011 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-sysctl-d\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947030 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e729724-d310-4884-a2d0-54af5308124c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947044 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-run-systemd\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947068 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947084 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-run-systemd\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947091 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-sys-fs\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947103 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-sysctl-d\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947114 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947119 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-var-lib-openvswitch\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947142 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-sysconfig\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947148 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/52432f95-9eb4-4fcd-b795-e475f3399c62-sys-fs\") pod \"aws-ebs-csi-driver-node-xz7t7\" (UID: \"52432f95-9eb4-4fcd-b795-e475f3399c62\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xz7t7" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947169 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-etc-kubernetes\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947180 4910 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947214 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-etc-kubernetes\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947194 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-sysconfig\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947167 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-var-lib-openvswitch\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947230 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-run\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947190 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-run\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947271 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e729724-d310-4884-a2d0-54af5308124c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947303 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-kubelet\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947331 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-cni-bin\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947356 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47cff5a0-505b-45e9-9788-cbe4b557917e-system-cni-dir\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947368 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-kubelet\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947376 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-hostroot\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947379 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-cni-bin\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947397 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47cff5a0-505b-45e9-9788-cbe4b557917e-system-cni-dir\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947410 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-systemd\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947418 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-hostroot\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947438 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-var-lib-cni-bin\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947465 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947493 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-var-lib-cni-bin\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947492 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-cnibin\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947467 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-systemd\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947515 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947523 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9255a1d5-2514-441e-b942-6c39843e38f8-dbus\") pod \"global-pull-secret-syncer-lhtkb\" (UID: \"9255a1d5-2514-441e-b942-6c39843e38f8\") " pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947531 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-cnibin\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947547 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947569 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-run-netns\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947588 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9255a1d5-2514-441e-b942-6c39843e38f8-dbus\") pod \"global-pull-secret-syncer-lhtkb\" (UID: \"9255a1d5-2514-441e-b942-6c39843e38f8\") " pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947591 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-modprobe-d\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947623 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2552d079-24ff-4651-b77e-155af2fe48e9-host-slash\") pod \"iptables-alerter-qn24x\" (UID: \"2552d079-24ff-4651-b77e-155af2fe48e9\") " pod="openshift-network-operator/iptables-alerter-qn24x" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947656 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-etc-modprobe-d\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947653 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-d4kds\" (UID: \"4cd15118-0a59-4122-bdf1-56710c463262\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947685 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-systemd-units\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:26.947696 4910 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:26.947715 4910 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947727 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-run-netns\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947738 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e729724-d310-4884-a2d0-54af5308124c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947782 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-systemd-units\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947744 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2552d079-24ff-4651-b77e-155af2fe48e9-host-slash\") pod \"iptables-alerter-qn24x\" (UID: \"2552d079-24ff-4651-b77e-155af2fe48e9\") " pod="openshift-network-operator/iptables-alerter-qn24x" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:26.947748 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs podName:70c870bc-c9e6-4d7e-bc99-ba556ce31cde nodeName:}" failed. No retries permitted until 2026-10-31 19:19:27.447733281 +0000 UTC m=+1.810553634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs") pod "router-default-8c75d8d7f-tzzdl" (UID: "70c870bc-c9e6-4d7e-bc99-ba556ce31cde") : secret "router-metrics-certs-default" not found Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:26.947841 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls podName:4cd15118-0a59-4122-bdf1-56710c463262 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:27.447825603 +0000 UTC m=+1.810645955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-d4kds" (UID: "4cd15118-0a59-4122-bdf1-56710c463262") : secret "cluster-monitoring-operator-tls" not found Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947859 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-run-netns\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947889 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-host-run-netns\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947888 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-etc-openvswitch\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947928 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9q7w4\" (UID: \"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947938 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-etc-openvswitch\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947953 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-var-lib-kubelet\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947985 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-var-lib-kubelet\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.947983 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9255a1d5-2514-441e-b942-6c39843e38f8-kubelet-config\") pod \"global-pull-secret-syncer-lhtkb\" (UID: \"9255a1d5-2514-441e-b942-6c39843e38f8\") " pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:26.948001 4910 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.948015 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-run-openvswitch\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.948028 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9255a1d5-2514-441e-b942-6c39843e38f8-kubelet-config\") pod \"global-pull-secret-syncer-lhtkb\" (UID: \"9255a1d5-2514-441e-b942-6c39843e38f8\") " pod="kube-system/global-pull-secret-syncer-lhtkb" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:26.948039 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls podName:f4d13f0a-4f53-4659-9a4e-0e1020f98b2e nodeName:}" failed. No retries permitted until 2026-10-31 19:19:27.448028076 +0000 UTC m=+1.810848427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-9q7w4" (UID: "f4d13f0a-4f53-4659-9a4e-0e1020f98b2e") : secret "samples-operator-tls" not found Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.948041 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81405623-de29-4fc6-bccf-447967669d89-run-openvswitch\") pod \"ovnkube-node-m2pjd\" (UID: \"81405623-de29-4fc6-bccf-447967669d89\") " pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.948055 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47cff5a0-505b-45e9-9788-cbe4b557917e-cnibin\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.948084 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-sys\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.948106 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47cff5a0-505b-45e9-9788-cbe4b557917e-cnibin\") pod \"multus-additional-cni-plugins-bhqht\" (UID: \"47cff5a0-505b-45e9-9788-cbe4b557917e\") " pod="openshift-multus/multus-additional-cni-plugins-bhqht" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.948109 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-var-lib-kubelet\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.948141 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5708be73-8282-4caa-bce3-46d4b708860e-host-var-lib-kubelet\") pod \"multus-tm5fh\" (UID: \"5708be73-8282-4caa-bce3-46d4b708860e\") " pod="openshift-multus/multus-tm5fh" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.948146 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf30e3e5-bfd9-4f93-9fe9-1514562c4267-sys\") pod \"tuned-4ngtp\" (UID: \"bf30e3e5-bfd9-4f93-9fe9-1514562c4267\") " pod="openshift-cluster-node-tuning-operator/tuned-4ngtp" Oct 31 19:19:26.948393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.948149 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5418e7b8-1471-4aa5-ad70-5d60b87c572a-metrics-tls\") pod \"dns-default-5nm2g\" (UID: \"5418e7b8-1471-4aa5-ad70-5d60b87c572a\") " pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:26.951113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.949459 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e729724-d310-4884-a2d0-54af5308124c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5nkwp\" (UID: \"4e729724-d310-4884-a2d0-54af5308124c\") " pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:26.951113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.949741 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88a1fa4b-1191-434a-8e7c-783163ece24d-metrics-certs\") pod \"network-metrics-daemon-dfbrp\" (UID: \"88a1fa4b-1191-434a-8e7c-783163ece24d\") " pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:19:26.951113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:26.950127 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5418e7b8-1471-4aa5-ad70-5d60b87c572a-metrics-tls\") pod \"dns-default-5nm2g\" (UID: \"5418e7b8-1471-4aa5-ad70-5d60b87c572a\") " pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:27.018768 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.018734 4910 scope.go:122] "RemoveContainer" containerID="3a9fb18642a3e4ea0da071d9b3009cfddbd0e22c650b9a07b4a1d921bd019368" Oct 31 19:19:27.028374 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.028349 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dfbrp" Oct 31 19:19:27.029727 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.029709 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:27.031850 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.031833 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5nkwp" Oct 31 19:19:27.119970 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.113396 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:27.330969 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.330907 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dfbrp"] Oct 31 19:19:27.333268 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:27.333240 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a1fa4b_1191_434a_8e7c_783163ece24d.slice/crio-9bec2e2a5864d0803c39f976b17ddb26f44db7a7bf6e711152f5b94f17b1a5db WatchSource:0}: Error finding container 9bec2e2a5864d0803c39f976b17ddb26f44db7a7bf6e711152f5b94f17b1a5db: Status 404 returned error can't find the container with id 9bec2e2a5864d0803c39f976b17ddb26f44db7a7bf6e711152f5b94f17b1a5db Oct 31 19:19:27.335238 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.335223 4910 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 19:19:27.374938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.374902 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5nm2g"] Oct 31 19:19:27.377186 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:27.377161 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5418e7b8_1471_4aa5_ad70_5d60b87c572a.slice/crio-9f14f919d3b778941be947f7c22387ab39bce6c3b8348275fbab4e91603f194d WatchSource:0}: Error finding container 9f14f919d3b778941be947f7c22387ab39bce6c3b8348275fbab4e91603f194d: Status 404 returned error can't find the container with id 9f14f919d3b778941be947f7c22387ab39bce6c3b8348275fbab4e91603f194d Oct 31 19:19:27.409866 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.409845 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5nkwp"] Oct 31 19:19:27.411427 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:27.411399 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e729724_d310_4884_a2d0_54af5308124c.slice/crio-74eb651fdb0c62e9cdc3d41e5163cdf3a214b14156660af8913f61bd4c148ba5 WatchSource:0}: Error finding container 74eb651fdb0c62e9cdc3d41e5163cdf3a214b14156660af8913f61bd4c148ba5: Status 404 returned error can't find the container with id 74eb651fdb0c62e9cdc3d41e5163cdf3a214b14156660af8913f61bd4c148ba5 Oct 31 19:19:27.466346 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.466316 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:27.466426 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.466358 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-d4kds\" (UID: \"4cd15118-0a59-4122-bdf1-56710c463262\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" Oct 31 19:19:27.466426 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.466392 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9q7w4\" (UID: \"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:27.466535 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.466435 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:27.466535 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:27.466459 4910 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 19:19:27.466535 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:27.466503 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs podName:70c870bc-c9e6-4d7e-bc99-ba556ce31cde nodeName:}" failed. No retries permitted until 2026-10-31 19:19:28.466490355 +0000 UTC m=+2.829310708 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs") pod "router-default-8c75d8d7f-tzzdl" (UID: "70c870bc-c9e6-4d7e-bc99-ba556ce31cde") : secret "router-metrics-certs-default" not found Oct 31 19:19:27.466535 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:27.466530 4910 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 19:19:27.466701 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:27.466535 4910 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 19:19:27.466701 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.466543 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:27.466701 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:27.466581 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls podName:f4d13f0a-4f53-4659-9a4e-0e1020f98b2e nodeName:}" failed. No retries permitted until 2026-10-31 19:19:28.466564923 +0000 UTC m=+2.829385277 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-9q7w4" (UID: "f4d13f0a-4f53-4659-9a4e-0e1020f98b2e") : secret "samples-operator-tls" not found Oct 31 19:19:27.466701 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:27.466600 4910 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 19:19:27.466701 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:27.466608 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls podName:4cd15118-0a59-4122-bdf1-56710c463262 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:28.466591789 +0000 UTC m=+2.829412157 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-d4kds" (UID: "4cd15118-0a59-4122-bdf1-56710c463262") : secret "cluster-monitoring-operator-tls" not found Oct 31 19:19:27.466701 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:27.466612 4910 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8ccd4bc6-xlml9: secret "image-registry-tls" not found Oct 31 19:19:27.466701 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:27.466644 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle podName:70c870bc-c9e6-4d7e-bc99-ba556ce31cde nodeName:}" failed. No retries permitted until 2026-10-31 19:19:28.466633387 +0000 UTC m=+2.829453737 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle") pod "router-default-8c75d8d7f-tzzdl" (UID: "70c870bc-c9e6-4d7e-bc99-ba556ce31cde") : configmap references non-existent config key: service-ca.crt Oct 31 19:19:27.466701 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:27.466657 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls podName:3a063ed7-8ce0-414e-8bbf-7733c464b507 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:28.466651635 +0000 UTC m=+2.829471985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls") pod "image-registry-5c8ccd4bc6-xlml9" (UID: "3a063ed7-8ce0-414e-8bbf-7733c464b507") : secret "image-registry-tls" not found Oct 31 19:19:27.466701 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.466668 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:27.466701 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.466694 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-28mzh\" (UID: \"9cf7d7f5-3936-45b3-ba2c-597cf99cfb89\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-28mzh" Oct 31 19:19:27.467010 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:27.466746 4910 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 19:19:27.467010 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:27.466755 4910 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f878896cc-hzn9v: secret "image-registry-tls" not found Oct 31 19:19:27.467010 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:27.466785 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls podName:eb1a9bea-399b-4281-9ada-a232790cc713 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:28.466775013 +0000 UTC m=+2.829595376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls") pod "image-registry-6f878896cc-hzn9v" (UID: "eb1a9bea-399b-4281-9ada-a232790cc713") : secret "image-registry-tls" not found Oct 31 19:19:27.467010 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:27.466799 4910 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 19:19:27.467010 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:27.466830 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert podName:9cf7d7f5-3936-45b3-ba2c-597cf99cfb89 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:28.466823369 +0000 UTC m=+2.829643719 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-28mzh" (UID: "9cf7d7f5-3936-45b3-ba2c-597cf99cfb89") : secret "networking-console-plugin-cert" not found Oct 31 19:19:27.811928 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.811889 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 19:19:27.812349 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.812319 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/1.log" Oct 31 19:19:27.812393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.812348 4910 generic.go:356] "Generic (PLEG): container finished" podID="76e07820-d888-4138-a41a-59ff4565accd" containerID="610d3dfe5afb2704670f6c0d8e9751e288a77c4f23c4e6f53260bd910268aa7b" exitCode=255 Oct 31 19:19:27.812478 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.812446 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" event={"ID":"76e07820-d888-4138-a41a-59ff4565accd","Type":"ContainerDied","Data":"610d3dfe5afb2704670f6c0d8e9751e288a77c4f23c4e6f53260bd910268aa7b"} Oct 31 19:19:27.812610 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.812509 4910 scope.go:122] "RemoveContainer" containerID="3a9fb18642a3e4ea0da071d9b3009cfddbd0e22c650b9a07b4a1d921bd019368" Oct 31 19:19:27.812756 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.812736 4910 scope.go:122] "RemoveContainer" containerID="610d3dfe5afb2704670f6c0d8e9751e288a77c4f23c4e6f53260bd910268aa7b" Oct 31 19:19:27.812957 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:27.812940 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6vpwc_openshift-console-operator(76e07820-d888-4138-a41a-59ff4565accd)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" podUID="76e07820-d888-4138-a41a-59ff4565accd" Oct 31 19:19:27.814091 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.814069 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5nkwp" event={"ID":"4e729724-d310-4884-a2d0-54af5308124c","Type":"ContainerStarted","Data":"d8f8d48ece85bd20add951a7c193485f41aa2c7e32e1f971e3cd58291e86e48c"} Oct 31 19:19:27.814165 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.814123 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5nkwp" event={"ID":"4e729724-d310-4884-a2d0-54af5308124c","Type":"ContainerStarted","Data":"74eb651fdb0c62e9cdc3d41e5163cdf3a214b14156660af8913f61bd4c148ba5"} Oct 31 19:19:27.815184 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.815164 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5nm2g" event={"ID":"5418e7b8-1471-4aa5-ad70-5d60b87c572a","Type":"ContainerStarted","Data":"9f14f919d3b778941be947f7c22387ab39bce6c3b8348275fbab4e91603f194d"} Oct 31 19:19:27.816287 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:27.816265 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dfbrp" event={"ID":"88a1fa4b-1191-434a-8e7c-783163ece24d","Type":"ContainerStarted","Data":"9bec2e2a5864d0803c39f976b17ddb26f44db7a7bf6e711152f5b94f17b1a5db"} Oct 31 19:19:28.194256 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:28.194191 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-85mq2_28c13795-a1de-4761-b3fc-feef76ac730d/dns-node-resolver/0.log" Oct 31 19:19:28.336616 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:28.336589 4910 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:28.477148 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:28.477063 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:28.477148 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:28.477123 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-d4kds\" (UID: \"4cd15118-0a59-4122-bdf1-56710c463262\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" Oct 31 19:19:28.477371 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:28.477160 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9q7w4\" (UID: \"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:28.477371 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:28.477210 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:28.477371 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:28.477242 4910 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 19:19:28.477371 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:28.477312 4910 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 19:19:28.477371 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:28.477250 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:28.477371 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:28.477316 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs podName:70c870bc-c9e6-4d7e-bc99-ba556ce31cde nodeName:}" failed. No retries permitted until 2026-10-31 19:19:30.477298595 +0000 UTC m=+4.840118947 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs") pod "router-default-8c75d8d7f-tzzdl" (UID: "70c870bc-c9e6-4d7e-bc99-ba556ce31cde") : secret "router-metrics-certs-default" not found Oct 31 19:19:28.477371 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:28.477312 4910 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 19:19:28.477711 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:28.477365 4910 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 19:19:28.477711 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:28.477411 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle podName:70c870bc-c9e6-4d7e-bc99-ba556ce31cde nodeName:}" failed. No retries permitted until 2026-10-31 19:19:30.477392026 +0000 UTC m=+4.840212380 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle") pod "router-default-8c75d8d7f-tzzdl" (UID: "70c870bc-c9e6-4d7e-bc99-ba556ce31cde") : configmap references non-existent config key: service-ca.crt Oct 31 19:19:28.477711 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:28.477426 4910 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8ccd4bc6-xlml9: secret "image-registry-tls" not found Oct 31 19:19:28.477711 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:28.477443 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls podName:4cd15118-0a59-4122-bdf1-56710c463262 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:30.477432838 +0000 UTC m=+4.840253192 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-d4kds" (UID: "4cd15118-0a59-4122-bdf1-56710c463262") : secret "cluster-monitoring-operator-tls" not found Oct 31 19:19:28.477711 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:28.477464 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls podName:f4d13f0a-4f53-4659-9a4e-0e1020f98b2e nodeName:}" failed. No retries permitted until 2026-10-31 19:19:30.477456178 +0000 UTC m=+4.840276528 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-9q7w4" (UID: "f4d13f0a-4f53-4659-9a4e-0e1020f98b2e") : secret "samples-operator-tls" not found Oct 31 19:19:28.477711 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:28.477505 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:28.477711 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:28.477515 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls podName:3a063ed7-8ce0-414e-8bbf-7733c464b507 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:30.477495984 +0000 UTC m=+4.840316355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls") pod "image-registry-5c8ccd4bc6-xlml9" (UID: "3a063ed7-8ce0-414e-8bbf-7733c464b507") : secret "image-registry-tls" not found Oct 31 19:19:28.477711 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:28.477568 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-28mzh\" (UID: \"9cf7d7f5-3936-45b3-ba2c-597cf99cfb89\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-28mzh" Oct 31 19:19:28.477711 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:28.477582 4910 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 19:19:28.477711 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:28.477594 4910 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f878896cc-hzn9v: secret "image-registry-tls" not found Oct 31 19:19:28.477711 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:28.477628 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls podName:eb1a9bea-399b-4281-9ada-a232790cc713 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:30.477618056 +0000 UTC m=+4.840438441 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls") pod "image-registry-6f878896cc-hzn9v" (UID: "eb1a9bea-399b-4281-9ada-a232790cc713") : secret "image-registry-tls" not found Oct 31 19:19:28.477711 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:28.477644 4910 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 19:19:28.477711 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:28.477684 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert podName:9cf7d7f5-3936-45b3-ba2c-597cf99cfb89 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:30.477674731 +0000 UTC m=+4.840495085 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-28mzh" (UID: "9cf7d7f5-3936-45b3-ba2c-597cf99cfb89") : secret "networking-console-plugin-cert" not found Oct 31 19:19:28.822214 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:28.822181 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5nkwp" event={"ID":"4e729724-d310-4884-a2d0-54af5308124c","Type":"ContainerStarted","Data":"da092a6aad5702687d6df19e6acca083b2d5c45889694410f0cc14a68b923a8d"} Oct 31 19:19:28.824207 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:28.824179 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dfbrp" event={"ID":"88a1fa4b-1191-434a-8e7c-783163ece24d","Type":"ContainerStarted","Data":"a9f2d5cd68fdbac36192013d596cd80c3d38a8a9352773a6ce5e35d8c93fe580"} Oct 31 19:19:28.824344 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:28.824243 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dfbrp" event={"ID":"88a1fa4b-1191-434a-8e7c-783163ece24d","Type":"ContainerStarted","Data":"4d3d27b05c55cfae70c7fb79370e83cd645ca95bf1f7a59c515a44b05ac43829"} Oct 31 19:19:28.826156 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:28.826133 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 19:19:28.826459 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:28.826434 4910 scope.go:122] "RemoveContainer" containerID="610d3dfe5afb2704670f6c0d8e9751e288a77c4f23c4e6f53260bd910268aa7b" Oct 31 19:19:28.826656 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:28.826634 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6vpwc_openshift-console-operator(76e07820-d888-4138-a41a-59ff4565accd)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" podUID="76e07820-d888-4138-a41a-59ff4565accd" Oct 31 19:19:29.394801 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:29.394729 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8m482_84f5548f-7d70-4391-89c9-3bd3c1784717/node-ca/0.log" Oct 31 19:19:29.831446 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:29.831414 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5nm2g" event={"ID":"5418e7b8-1471-4aa5-ad70-5d60b87c572a","Type":"ContainerStarted","Data":"c6363e0fe67a2fdd2a0af6442bec3cb365412c52af647517dff2887bdeaf5305"} Oct 31 19:19:29.831943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:29.831475 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5nm2g" event={"ID":"5418e7b8-1471-4aa5-ad70-5d60b87c572a","Type":"ContainerStarted","Data":"473abdb5762650b0e07e2b2c84415757a70c4c4cfa995de2ec1f9a25b12e1d26"} Oct 31 19:19:29.831943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:29.831767 4910 scope.go:122] "RemoveContainer" containerID="610d3dfe5afb2704670f6c0d8e9751e288a77c4f23c4e6f53260bd910268aa7b" Oct 31 19:19:29.832131 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:29.831969 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6vpwc_openshift-console-operator(76e07820-d888-4138-a41a-59ff4565accd)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" podUID="76e07820-d888-4138-a41a-59ff4565accd" Oct 31 19:19:29.994026 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:29.993995 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q8j58_92aaddbc-6268-443c-b3bd-9f74ab6b9a4b/serve-healthcheck-canary/0.log" Oct 31 19:19:30.194435 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:30.194355 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-4fsp7_7bce545d-6640-49e8-848c-aa9f8db19902/migrator/0.log" Oct 31 19:19:30.342251 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:30.342226 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:30.497338 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:30.497248 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9q7w4\" (UID: \"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:30.497503 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:30.497368 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:30.497503 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:30.497420 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:30.497503 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:30.497440 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:30.497503 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:30.497460 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-28mzh\" (UID: \"9cf7d7f5-3936-45b3-ba2c-597cf99cfb89\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-28mzh" Oct 31 19:19:30.497503 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:30.497473 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle podName:70c870bc-c9e6-4d7e-bc99-ba556ce31cde nodeName:}" failed. No retries permitted until 2026-10-31 19:19:34.497457544 +0000 UTC m=+8.860277897 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle") pod "router-default-8c75d8d7f-tzzdl" (UID: "70c870bc-c9e6-4d7e-bc99-ba556ce31cde") : configmap references non-existent config key: service-ca.crt Oct 31 19:19:30.497715 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:30.497517 4910 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 19:19:30.497715 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:30.497543 4910 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 19:19:30.497715 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:30.497560 4910 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8ccd4bc6-xlml9: secret "image-registry-tls" not found Oct 31 19:19:30.497715 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:30.497574 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls podName:f4d13f0a-4f53-4659-9a4e-0e1020f98b2e nodeName:}" failed. No retries permitted until 2026-10-31 19:19:34.497558482 +0000 UTC m=+8.860378834 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-9q7w4" (UID: "f4d13f0a-4f53-4659-9a4e-0e1020f98b2e") : secret "samples-operator-tls" not found Oct 31 19:19:30.497715 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:30.497605 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls podName:3a063ed7-8ce0-414e-8bbf-7733c464b507 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:34.497592536 +0000 UTC m=+8.860412899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls") pod "image-registry-5c8ccd4bc6-xlml9" (UID: "3a063ed7-8ce0-414e-8bbf-7733c464b507") : secret "image-registry-tls" not found Oct 31 19:19:30.497715 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:30.497604 4910 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 19:19:30.497715 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:30.497627 4910 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f878896cc-hzn9v: secret "image-registry-tls" not found Oct 31 19:19:30.497715 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:30.497628 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:30.497715 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:30.497669 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls podName:eb1a9bea-399b-4281-9ada-a232790cc713 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:34.497654598 +0000 UTC m=+8.860474956 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls") pod "image-registry-6f878896cc-hzn9v" (UID: "eb1a9bea-399b-4281-9ada-a232790cc713") : secret "image-registry-tls" not found Oct 31 19:19:30.497715 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:30.497690 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-d4kds\" (UID: \"4cd15118-0a59-4122-bdf1-56710c463262\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" Oct 31 19:19:30.500055 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:30.500033 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-metrics-certs\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:30.500165 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:30.500082 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9cf7d7f5-3936-45b3-ba2c-597cf99cfb89-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-28mzh\" (UID: \"9cf7d7f5-3936-45b3-ba2c-597cf99cfb89\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-28mzh" Oct 31 19:19:30.500165 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:30.500130 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4cd15118-0a59-4122-bdf1-56710c463262-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-d4kds\" (UID: \"4cd15118-0a59-4122-bdf1-56710c463262\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" Oct 31 19:19:30.597434 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:30.597397 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-58hlm_ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f/kube-storage-version-migrator-operator/0.log" Oct 31 19:19:30.619056 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:30.619020 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" Oct 31 19:19:30.621212 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:30.621183 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-28mzh" Oct 31 19:19:30.702499 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:30.702473 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:30.836981 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:30.836929 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5nkwp" event={"ID":"4e729724-d310-4884-a2d0-54af5308124c","Type":"ContainerStarted","Data":"01b567a1ebf26e4ddf9f48326506ad3255b59deb0123338ac60ff4239705b250"} Oct 31 19:19:30.837375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:30.837048 4910 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 19:19:30.837577 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:30.837558 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:30.893535 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:30.893508 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-28mzh"] Oct 31 19:19:30.896057 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:30.896032 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds"] Oct 31 19:19:30.896226 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:30.896205 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf7d7f5_3936_45b3_ba2c_597cf99cfb89.slice/crio-c6fbae2e2cc65c128cbc90fbe016f40dc8420c185d39024b62fb565661009fb6 WatchSource:0}: Error finding container c6fbae2e2cc65c128cbc90fbe016f40dc8420c185d39024b62fb565661009fb6: Status 404 returned error can't find the container with id c6fbae2e2cc65c128cbc90fbe016f40dc8420c185d39024b62fb565661009fb6 Oct 31 19:19:31.842511 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:31.842423 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" event={"ID":"4cd15118-0a59-4122-bdf1-56710c463262","Type":"ContainerStarted","Data":"bbd141d52d1668578e663cf54cdd9a877c5296339b71a1bc1b3160160d7b7520"} Oct 31 19:19:31.844774 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:31.844749 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-28mzh" event={"ID":"9cf7d7f5-3936-45b3-ba2c-597cf99cfb89","Type":"ContainerStarted","Data":"c6fbae2e2cc65c128cbc90fbe016f40dc8420c185d39024b62fb565661009fb6"} Oct 31 19:19:32.852473 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:32.852435 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-28mzh" event={"ID":"9cf7d7f5-3936-45b3-ba2c-597cf99cfb89","Type":"ContainerStarted","Data":"fb3abd96c1603defd4ff236cdedef3df8e25d939b55c630c80b0de2d7b11aa0b"} Oct 31 19:19:32.854270 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:32.854231 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-d4kds" event={"ID":"4cd15118-0a59-4122-bdf1-56710c463262","Type":"ContainerStarted","Data":"31c16518390c72c7263811fb88cc391acf6988770344f1afbd62f482e08e6dcb"} Oct 31 19:19:32.942840 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:32.942768 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:19:34.539658 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:34.539629 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9q7w4\" (UID: \"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:34.540111 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:34.539674 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:34.540111 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:34.539703 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:34.540111 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:34.539729 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:34.540111 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:34.539820 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle podName:70c870bc-c9e6-4d7e-bc99-ba556ce31cde nodeName:}" failed. No retries permitted until 2026-10-31 19:19:42.539802056 +0000 UTC m=+16.902622423 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle") pod "router-default-8c75d8d7f-tzzdl" (UID: "70c870bc-c9e6-4d7e-bc99-ba556ce31cde") : configmap references non-existent config key: service-ca.crt Oct 31 19:19:34.540111 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:34.539844 4910 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 19:19:34.540111 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:34.539856 4910 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f878896cc-hzn9v: secret "image-registry-tls" not found Oct 31 19:19:34.540111 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:34.539872 4910 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 19:19:34.540111 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:34.539894 4910 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5c8ccd4bc6-xlml9: secret "image-registry-tls" not found Oct 31 19:19:34.540111 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:34.539897 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls podName:eb1a9bea-399b-4281-9ada-a232790cc713 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:42.539886682 +0000 UTC m=+16.902707036 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls") pod "image-registry-6f878896cc-hzn9v" (UID: "eb1a9bea-399b-4281-9ada-a232790cc713") : secret "image-registry-tls" not found Oct 31 19:19:34.540111 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:34.539966 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls podName:3a063ed7-8ce0-414e-8bbf-7733c464b507 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:42.539948655 +0000 UTC m=+16.902769022 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls") pod "image-registry-5c8ccd4bc6-xlml9" (UID: "3a063ed7-8ce0-414e-8bbf-7733c464b507") : secret "image-registry-tls" not found Oct 31 19:19:34.542144 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:34.542117 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f4d13f0a-4f53-4659-9a4e-0e1020f98b2e-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9q7w4\" (UID: \"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:34.548478 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:34.548457 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m2pjd" Oct 31 19:19:34.819665 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:34.819635 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" Oct 31 19:19:35.089766 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:35.089692 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4"] Oct 31 19:19:35.763213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:35.763178 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wxdtf" Oct 31 19:19:35.865864 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:35.865817 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" event={"ID":"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e","Type":"ContainerStarted","Data":"a4a58c82c1269584de5d4f59aa22c6b5b99b124d4b27bd5ec365955bd5eecee7"} Oct 31 19:19:36.874433 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:36.874391 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" event={"ID":"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e","Type":"ContainerStarted","Data":"33fdc1132f3be3fa64d60638e924a1489d39df543604f1a5469c47c926a2a956"} Oct 31 19:19:36.874802 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:36.874447 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9q7w4" event={"ID":"f4d13f0a-4f53-4659-9a4e-0e1020f98b2e","Type":"ContainerStarted","Data":"bdb5c3a819a3bf1f9ebc827d20412827316c147cd4eaae357ea8c2ed3e01e646"} Oct 31 19:19:37.113444 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:37.113406 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:37.113830 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:37.113812 4910 scope.go:122] "RemoveContainer" containerID="610d3dfe5afb2704670f6c0d8e9751e288a77c4f23c4e6f53260bd910268aa7b" Oct 31 19:19:37.114071 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:37.114048 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-6vpwc_openshift-console-operator(76e07820-d888-4138-a41a-59ff4565accd)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" podUID="76e07820-d888-4138-a41a-59ff4565accd" Oct 31 19:19:37.881326 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:37.881302 4910 scope.go:122] "RemoveContainer" containerID="610d3dfe5afb2704670f6c0d8e9751e288a77c4f23c4e6f53260bd910268aa7b" Oct 31 19:19:38.886090 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:38.886064 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 19:19:38.886510 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:38.886146 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" event={"ID":"76e07820-d888-4138-a41a-59ff4565accd","Type":"ContainerStarted","Data":"25dc0b10e70158a564aff78841f0ae23a03aee130a057a3e98a003763aa72f04"} Oct 31 19:19:38.886510 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:38.886443 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:38.891415 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:38.891392 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-6vpwc" Oct 31 19:19:40.847562 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:40.847532 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5nm2g" Oct 31 19:19:42.614087 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:42.614048 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:42.614562 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:42.614094 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:42.614562 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:42.614126 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:42.614792 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:42.614769 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c870bc-c9e6-4d7e-bc99-ba556ce31cde-service-ca-bundle\") pod \"router-default-8c75d8d7f-tzzdl\" (UID: \"70c870bc-c9e6-4d7e-bc99-ba556ce31cde\") " pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:42.616475 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:42.616447 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls\") pod \"image-registry-5c8ccd4bc6-xlml9\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:42.616563 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:42.616524 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls\") pod \"image-registry-6f878896cc-hzn9v\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:42.620445 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:42.620430 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:42.620580 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:42.620562 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:42.627538 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:42.627518 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:42.723818 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:19:42.723599 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:19:42.906693 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:42.906672 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f878896cc-hzn9v"] Oct 31 19:19:42.908705 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:42.908677 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1a9bea_399b_4281_9ada_a232790cc713.slice/crio-037095a0c0b2f890ad51cc5209eecbeb839d4af6601e1166cb6114eb431bc8c6 WatchSource:0}: Error finding container 037095a0c0b2f890ad51cc5209eecbeb839d4af6601e1166cb6114eb431bc8c6: Status 404 returned error can't find the container with id 037095a0c0b2f890ad51cc5209eecbeb839d4af6601e1166cb6114eb431bc8c6 Oct 31 19:19:42.911078 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:42.911059 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8c75d8d7f-tzzdl"] Oct 31 19:19:42.913243 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:42.913219 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c870bc_c9e6_4d7e_bc99_ba556ce31cde.slice/crio-ee67c0bd76bbbc3ac30f4f64e8545ca431e9ae8d9f9dd00b7b400a7d1dd0b02b WatchSource:0}: Error finding container ee67c0bd76bbbc3ac30f4f64e8545ca431e9ae8d9f9dd00b7b400a7d1dd0b02b: Status 404 returned error can't find the container with id ee67c0bd76bbbc3ac30f4f64e8545ca431e9ae8d9f9dd00b7b400a7d1dd0b02b Oct 31 19:19:42.915987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:42.915963 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c8ccd4bc6-xlml9"] Oct 31 19:19:42.918686 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:42.918665 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a063ed7_8ce0_414e_8bbf_7733c464b507.slice/crio-8cd2179a0d8b1d9a0419e6632a759dcbf016b2a7906b028f78f71ed9601b2137 WatchSource:0}: Error finding container 8cd2179a0d8b1d9a0419e6632a759dcbf016b2a7906b028f78f71ed9601b2137: Status 404 returned error can't find the container with id 8cd2179a0d8b1d9a0419e6632a759dcbf016b2a7906b028f78f71ed9601b2137 Oct 31 19:19:43.908390 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:43.908345 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" event={"ID":"70c870bc-c9e6-4d7e-bc99-ba556ce31cde","Type":"ContainerStarted","Data":"308cd233d032e9737bef08ab6d43d5b93441d443e14c7d0ba78d3a2b0ac387ac"} Oct 31 19:19:43.909007 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:43.908408 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" event={"ID":"70c870bc-c9e6-4d7e-bc99-ba556ce31cde","Type":"ContainerStarted","Data":"ee67c0bd76bbbc3ac30f4f64e8545ca431e9ae8d9f9dd00b7b400a7d1dd0b02b"} Oct 31 19:19:43.910121 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:43.910092 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" event={"ID":"3a063ed7-8ce0-414e-8bbf-7733c464b507","Type":"ContainerStarted","Data":"404a54de7acb449126e8312fe7ca34f921985eaa88f23f0615078a2d26b7c241"} Oct 31 19:19:43.910251 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:43.910135 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" event={"ID":"3a063ed7-8ce0-414e-8bbf-7733c464b507","Type":"ContainerStarted","Data":"8cd2179a0d8b1d9a0419e6632a759dcbf016b2a7906b028f78f71ed9601b2137"} Oct 31 19:19:43.910251 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:43.910155 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:19:43.911822 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:43.911801 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" event={"ID":"eb1a9bea-399b-4281-9ada-a232790cc713","Type":"ContainerStarted","Data":"9841ef33f37bd9fd355a7b28e742814b05fe12775aeb6307aa67ef12cf8e55d4"} Oct 31 19:19:43.911970 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:43.911832 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" event={"ID":"eb1a9bea-399b-4281-9ada-a232790cc713","Type":"ContainerStarted","Data":"037095a0c0b2f890ad51cc5209eecbeb839d4af6601e1166cb6114eb431bc8c6"} Oct 31 19:19:43.911970 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:43.911926 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:19:44.009748 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:44.009707 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f878896cc-hzn9v"] Oct 31 19:19:44.621133 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:44.621103 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:44.623827 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:44.623802 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:44.921552 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:44.921463 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:44.922404 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:44.922385 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-8c75d8d7f-tzzdl" Oct 31 19:19:51.653069 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.653040 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-74jl5"] Oct 31 19:19:51.713515 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.713484 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.724290 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.724268 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 19:19:51.732303 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.732284 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 19:19:51.732380 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.732340 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mxbfw\"" Oct 31 19:19:51.740342 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.740320 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 19:19:51.740434 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.740418 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 19:19:51.779716 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.779687 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9543a93d-ce3e-4899-b961-68b45d88ff70-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.779859 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.779721 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9543a93d-ce3e-4899-b961-68b45d88ff70-metrics-client-ca\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.779859 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.779747 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9543a93d-ce3e-4899-b961-68b45d88ff70-root\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.779859 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.779841 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fr8p\" (UniqueName: \"kubernetes.io/projected/9543a93d-ce3e-4899-b961-68b45d88ff70-kube-api-access-7fr8p\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.779982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.779895 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9543a93d-ce3e-4899-b961-68b45d88ff70-sys\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.779982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.779967 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9543a93d-ce3e-4899-b961-68b45d88ff70-node-exporter-accelerators-collector-config\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.780049 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.779994 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9543a93d-ce3e-4899-b961-68b45d88ff70-node-exporter-textfile\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.780049 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.780024 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9543a93d-ce3e-4899-b961-68b45d88ff70-node-exporter-tls\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.780107 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.780070 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9543a93d-ce3e-4899-b961-68b45d88ff70-node-exporter-wtmp\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.880937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.880889 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9543a93d-ce3e-4899-b961-68b45d88ff70-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.881108 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.880960 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9543a93d-ce3e-4899-b961-68b45d88ff70-metrics-client-ca\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.881108 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.880989 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9543a93d-ce3e-4899-b961-68b45d88ff70-root\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.881108 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.881014 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fr8p\" (UniqueName: \"kubernetes.io/projected/9543a93d-ce3e-4899-b961-68b45d88ff70-kube-api-access-7fr8p\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.881263 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.881101 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9543a93d-ce3e-4899-b961-68b45d88ff70-root\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.881263 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.881148 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9543a93d-ce3e-4899-b961-68b45d88ff70-sys\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.881263 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.881202 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9543a93d-ce3e-4899-b961-68b45d88ff70-node-exporter-accelerators-collector-config\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.881263 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.881233 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9543a93d-ce3e-4899-b961-68b45d88ff70-node-exporter-textfile\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.881263 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.881237 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9543a93d-ce3e-4899-b961-68b45d88ff70-sys\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.881263 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.881255 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9543a93d-ce3e-4899-b961-68b45d88ff70-node-exporter-tls\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.881560 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.881319 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9543a93d-ce3e-4899-b961-68b45d88ff70-node-exporter-wtmp\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.881560 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.881469 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9543a93d-ce3e-4899-b961-68b45d88ff70-node-exporter-wtmp\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.881560 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.881543 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9543a93d-ce3e-4899-b961-68b45d88ff70-node-exporter-textfile\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.881726 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.881565 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9543a93d-ce3e-4899-b961-68b45d88ff70-metrics-client-ca\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.881782 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.881745 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9543a93d-ce3e-4899-b961-68b45d88ff70-node-exporter-accelerators-collector-config\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.883255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.883234 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9543a93d-ce3e-4899-b961-68b45d88ff70-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.883423 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.883407 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9543a93d-ce3e-4899-b961-68b45d88ff70-node-exporter-tls\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:51.901041 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:51.901013 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fr8p\" (UniqueName: \"kubernetes.io/projected/9543a93d-ce3e-4899-b961-68b45d88ff70-kube-api-access-7fr8p\") pod \"node-exporter-74jl5\" (UID: \"9543a93d-ce3e-4899-b961-68b45d88ff70\") " pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:52.065096 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:52.065063 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-74jl5" Oct 31 19:19:52.099713 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:19:52.099684 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9543a93d_ce3e_4899_b961_68b45d88ff70.slice/crio-aad8e42fef6d7902dfe420238e052480637735d495b6ccca1523a83747c23e64 WatchSource:0}: Error finding container aad8e42fef6d7902dfe420238e052480637735d495b6ccca1523a83747c23e64: Status 404 returned error can't find the container with id aad8e42fef6d7902dfe420238e052480637735d495b6ccca1523a83747c23e64 Oct 31 19:19:52.946092 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:52.946061 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5mkxz" Oct 31 19:19:52.950272 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:52.950244 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-74jl5" event={"ID":"9543a93d-ce3e-4899-b961-68b45d88ff70","Type":"ContainerStarted","Data":"aad8e42fef6d7902dfe420238e052480637735d495b6ccca1523a83747c23e64"} Oct 31 19:19:53.955728 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:53.955643 4910 generic.go:356] "Generic (PLEG): container finished" podID="9543a93d-ce3e-4899-b961-68b45d88ff70" containerID="79f2b1a00b14395f613384c17c135059c92d629da2d02d91daab7791563b07a2" exitCode=0 Oct 31 19:19:53.956114 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:53.955743 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-74jl5" event={"ID":"9543a93d-ce3e-4899-b961-68b45d88ff70","Type":"ContainerDied","Data":"79f2b1a00b14395f613384c17c135059c92d629da2d02d91daab7791563b07a2"} Oct 31 19:19:54.962493 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:54.962453 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-74jl5" event={"ID":"9543a93d-ce3e-4899-b961-68b45d88ff70","Type":"ContainerStarted","Data":"f63110682a120793165a513107fff1ed25bd232422d094c145e75e371db018d7"} Oct 31 19:19:54.962979 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:54.962566 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-74jl5" event={"ID":"9543a93d-ce3e-4899-b961-68b45d88ff70","Type":"ContainerStarted","Data":"5df2937d27d9fc769fc311c64f4eac3fb685fd68baa96780591101d086dc58f2"} Oct 31 19:19:54.988720 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:19:54.988671 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-74jl5" podStartSLOduration=2.399534712 podStartE2EDuration="3.988658073s" totalImagesPullingTime="1.510183941s" totalInitContainerRuntime="78.93942ms" isStatefulPod=true podCreationTimestamp="2026-10-31 19:19:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:54.986464026 +0000 UTC m=+29.349284398" watchObservedRunningTime="2026-10-31 19:19:54.988658073 +0000 UTC m=+29.351478442" Oct 31 19:20:04.925737 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:04.925709 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:20:05.930168 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:05.930091 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:20:06.822058 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:06.822018 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c8ccd4bc6-xlml9"] Oct 31 19:20:10.944344 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:10.944278 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" podUID="eb1a9bea-399b-4281-9ada-a232790cc713" containerName="registry" containerID="cri-o://9841ef33f37bd9fd355a7b28e742814b05fe12775aeb6307aa67ef12cf8e55d4" gracePeriod=30 Oct 31 19:20:11.192062 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:11.192040 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:20:11.251078 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:11.251012 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb1a9bea-399b-4281-9ada-a232790cc713-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb1a9bea-399b-4281-9ada-a232790cc713-registry-certificates\") pod \"eb1a9bea-399b-4281-9ada-a232790cc713\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " Oct 31 19:20:11.251078 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:11.251051 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb1a9bea-399b-4281-9ada-a232790cc713-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb1a9bea-399b-4281-9ada-a232790cc713-trusted-ca\") pod \"eb1a9bea-399b-4281-9ada-a232790cc713\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " Oct 31 19:20:11.251279 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:11.251091 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-kube-api-access-t4wnr\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-kube-api-access-t4wnr\") pod \"eb1a9bea-399b-4281-9ada-a232790cc713\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " Oct 31 19:20:11.251279 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:11.251121 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb1a9bea-399b-4281-9ada-a232790cc713-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb1a9bea-399b-4281-9ada-a232790cc713-installation-pull-secrets\") pod \"eb1a9bea-399b-4281-9ada-a232790cc713\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " Oct 31 19:20:11.251279 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:11.251164 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb1a9bea-399b-4281-9ada-a232790cc713-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb1a9bea-399b-4281-9ada-a232790cc713-ca-trust-extracted\") pod \"eb1a9bea-399b-4281-9ada-a232790cc713\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " Oct 31 19:20:11.251279 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:11.251207 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls\") pod \"eb1a9bea-399b-4281-9ada-a232790cc713\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " Oct 31 19:20:11.251279 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:11.251268 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb1a9bea-399b-4281-9ada-a232790cc713-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb1a9bea-399b-4281-9ada-a232790cc713-image-registry-private-configuration\") pod \"eb1a9bea-399b-4281-9ada-a232790cc713\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " Oct 31 19:20:11.251581 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:11.251287 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-bound-sa-token\") pod \"eb1a9bea-399b-4281-9ada-a232790cc713\" (UID: \"eb1a9bea-399b-4281-9ada-a232790cc713\") " Oct 31 19:20:11.251581 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:11.251461 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb1a9bea-399b-4281-9ada-a232790cc713-registry-certificates" pod "eb1a9bea-399b-4281-9ada-a232790cc713" (UID: "eb1a9bea-399b-4281-9ada-a232790cc713"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:20:11.251581 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:11.251565 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb1a9bea-399b-4281-9ada-a232790cc713-trusted-ca" pod "eb1a9bea-399b-4281-9ada-a232790cc713" (UID: "eb1a9bea-399b-4281-9ada-a232790cc713"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:20:11.253779 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:11.253746 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb1a9bea-399b-4281-9ada-a232790cc713-installation-pull-secrets" pod "eb1a9bea-399b-4281-9ada-a232790cc713" (UID: "eb1a9bea-399b-4281-9ada-a232790cc713"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:20:11.254060 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:11.254033 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-kube-api-access-t4wnr" pod "eb1a9bea-399b-4281-9ada-a232790cc713" (UID: "eb1a9bea-399b-4281-9ada-a232790cc713"). InnerVolumeSpecName "kube-api-access-t4wnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:20:11.254150 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:11.254117 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb1a9bea-399b-4281-9ada-a232790cc713-image-registry-private-configuration" pod "eb1a9bea-399b-4281-9ada-a232790cc713" (UID: "eb1a9bea-399b-4281-9ada-a232790cc713"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:20:11.254206 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:11.254139 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls" pod "eb1a9bea-399b-4281-9ada-a232790cc713" (UID: "eb1a9bea-399b-4281-9ada-a232790cc713"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:20:11.254206 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:11.254128 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-bound-sa-token" pod "eb1a9bea-399b-4281-9ada-a232790cc713" (UID: "eb1a9bea-399b-4281-9ada-a232790cc713"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:20:11.260300 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:11.260268 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1a9bea-399b-4281-9ada-a232790cc713-ca-trust-extracted" pod "eb1a9bea-399b-4281-9ada-a232790cc713" (UID: "eb1a9bea-399b-4281-9ada-a232790cc713"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:20:11.352463 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:11.352432 4910 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb1a9bea-399b-4281-9ada-a232790cc713-ca-trust-extracted\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:20:11.352463 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:11.352463 4910 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-registry-tls\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:20:11.352664 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:11.352478 4910 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb1a9bea-399b-4281-9ada-a232790cc713-image-registry-private-configuration\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:20:11.352664 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:11.352493 4910 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-bound-sa-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:20:11.352664 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:11.352506 4910 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb1a9bea-399b-4281-9ada-a232790cc713-registry-certificates\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:20:11.352664 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:11.352519 4910 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb1a9bea-399b-4281-9ada-a232790cc713-trusted-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:20:11.352664 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:11.352531 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4wnr\" (UniqueName: \"kubernetes.io/projected/eb1a9bea-399b-4281-9ada-a232790cc713-kube-api-access-t4wnr\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:20:11.352664 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:11.352544 4910 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb1a9bea-399b-4281-9ada-a232790cc713-installation-pull-secrets\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:20:12.025873 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:12.025775 4910 generic.go:356] "Generic (PLEG): container finished" podID="eb1a9bea-399b-4281-9ada-a232790cc713" containerID="9841ef33f37bd9fd355a7b28e742814b05fe12775aeb6307aa67ef12cf8e55d4" exitCode=0 Oct 31 19:20:12.025873 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:12.025816 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" Oct 31 19:20:12.025873 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:12.025848 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" event={"ID":"eb1a9bea-399b-4281-9ada-a232790cc713","Type":"ContainerDied","Data":"9841ef33f37bd9fd355a7b28e742814b05fe12775aeb6307aa67ef12cf8e55d4"} Oct 31 19:20:12.026459 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:12.025899 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f878896cc-hzn9v" event={"ID":"eb1a9bea-399b-4281-9ada-a232790cc713","Type":"ContainerDied","Data":"037095a0c0b2f890ad51cc5209eecbeb839d4af6601e1166cb6114eb431bc8c6"} Oct 31 19:20:12.026459 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:12.025947 4910 scope.go:122] "RemoveContainer" containerID="9841ef33f37bd9fd355a7b28e742814b05fe12775aeb6307aa67ef12cf8e55d4" Oct 31 19:20:12.034082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:12.034067 4910 scope.go:122] "RemoveContainer" containerID="9841ef33f37bd9fd355a7b28e742814b05fe12775aeb6307aa67ef12cf8e55d4" Oct 31 19:20:12.036864 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:20:12.036795 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9841ef33f37bd9fd355a7b28e742814b05fe12775aeb6307aa67ef12cf8e55d4\": container with ID starting with 9841ef33f37bd9fd355a7b28e742814b05fe12775aeb6307aa67ef12cf8e55d4 not found: ID does not exist" containerID="9841ef33f37bd9fd355a7b28e742814b05fe12775aeb6307aa67ef12cf8e55d4" Oct 31 19:20:12.036864 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:12.036837 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9841ef33f37bd9fd355a7b28e742814b05fe12775aeb6307aa67ef12cf8e55d4"} err="failed to get container status \"9841ef33f37bd9fd355a7b28e742814b05fe12775aeb6307aa67ef12cf8e55d4\": rpc error: code = NotFound desc = could not find container \"9841ef33f37bd9fd355a7b28e742814b05fe12775aeb6307aa67ef12cf8e55d4\": container with ID starting with 9841ef33f37bd9fd355a7b28e742814b05fe12775aeb6307aa67ef12cf8e55d4 not found: ID does not exist" Oct 31 19:20:12.047641 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:12.047620 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f878896cc-hzn9v"] Oct 31 19:20:12.053553 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:12.053533 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f878896cc-hzn9v"] Oct 31 19:20:12.712135 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:20:12.712098 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:20:13.752028 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:13.751994 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb1a9bea-399b-4281-9ada-a232790cc713" path="/var/lib/kubelet/pods/eb1a9bea-399b-4281-9ada-a232790cc713/volumes" Oct 31 19:20:31.841292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:31.841234 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" podUID="3a063ed7-8ce0-414e-8bbf-7733c464b507" containerName="registry" containerID="cri-o://404a54de7acb449126e8312fe7ca34f921985eaa88f23f0615078a2d26b7c241" gracePeriod=30 Oct 31 19:20:32.084527 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.084503 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:20:32.101286 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.101225 4910 generic.go:356] "Generic (PLEG): container finished" podID="3a063ed7-8ce0-414e-8bbf-7733c464b507" containerID="404a54de7acb449126e8312fe7ca34f921985eaa88f23f0615078a2d26b7c241" exitCode=0 Oct 31 19:20:32.101286 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.101261 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" event={"ID":"3a063ed7-8ce0-414e-8bbf-7733c464b507","Type":"ContainerDied","Data":"404a54de7acb449126e8312fe7ca34f921985eaa88f23f0615078a2d26b7c241"} Oct 31 19:20:32.101286 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.101273 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" Oct 31 19:20:32.101464 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.101300 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c8ccd4bc6-xlml9" event={"ID":"3a063ed7-8ce0-414e-8bbf-7733c464b507","Type":"ContainerDied","Data":"8cd2179a0d8b1d9a0419e6632a759dcbf016b2a7906b028f78f71ed9601b2137"} Oct 31 19:20:32.101464 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.101320 4910 scope.go:122] "RemoveContainer" containerID="404a54de7acb449126e8312fe7ca34f921985eaa88f23f0615078a2d26b7c241" Oct 31 19:20:32.109028 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.109002 4910 scope.go:122] "RemoveContainer" containerID="404a54de7acb449126e8312fe7ca34f921985eaa88f23f0615078a2d26b7c241" Oct 31 19:20:32.109284 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:20:32.109264 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"404a54de7acb449126e8312fe7ca34f921985eaa88f23f0615078a2d26b7c241\": container with ID starting with 404a54de7acb449126e8312fe7ca34f921985eaa88f23f0615078a2d26b7c241 not found: ID does not exist" containerID="404a54de7acb449126e8312fe7ca34f921985eaa88f23f0615078a2d26b7c241" Oct 31 19:20:32.109345 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.109298 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"404a54de7acb449126e8312fe7ca34f921985eaa88f23f0615078a2d26b7c241"} err="failed to get container status \"404a54de7acb449126e8312fe7ca34f921985eaa88f23f0615078a2d26b7c241\": rpc error: code = NotFound desc = could not find container \"404a54de7acb449126e8312fe7ca34f921985eaa88f23f0615078a2d26b7c241\": container with ID starting with 404a54de7acb449126e8312fe7ca34f921985eaa88f23f0615078a2d26b7c241 not found: ID does not exist" Oct 31 19:20:32.231759 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.231715 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a063ed7-8ce0-414e-8bbf-7733c464b507-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a063ed7-8ce0-414e-8bbf-7733c464b507-installation-pull-secrets\") pod \"3a063ed7-8ce0-414e-8bbf-7733c464b507\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " Oct 31 19:20:32.231968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.231794 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-bound-sa-token\") pod \"3a063ed7-8ce0-414e-8bbf-7733c464b507\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " Oct 31 19:20:32.231968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.231844 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a063ed7-8ce0-414e-8bbf-7733c464b507-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a063ed7-8ce0-414e-8bbf-7733c464b507-image-registry-private-configuration\") pod \"3a063ed7-8ce0-414e-8bbf-7733c464b507\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " Oct 31 19:20:32.231968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.231873 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a063ed7-8ce0-414e-8bbf-7733c464b507-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a063ed7-8ce0-414e-8bbf-7733c464b507-ca-trust-extracted\") pod \"3a063ed7-8ce0-414e-8bbf-7733c464b507\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " Oct 31 19:20:32.231968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.231934 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls\") pod \"3a063ed7-8ce0-414e-8bbf-7733c464b507\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " Oct 31 19:20:32.232220 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.231976 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-kube-api-access-4zsbj\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-kube-api-access-4zsbj\") pod \"3a063ed7-8ce0-414e-8bbf-7733c464b507\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " Oct 31 19:20:32.232220 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.232021 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a063ed7-8ce0-414e-8bbf-7733c464b507-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a063ed7-8ce0-414e-8bbf-7733c464b507-trusted-ca\") pod \"3a063ed7-8ce0-414e-8bbf-7733c464b507\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " Oct 31 19:20:32.232220 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.232046 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-certificates\") pod \"3a063ed7-8ce0-414e-8bbf-7733c464b507\" (UID: \"3a063ed7-8ce0-414e-8bbf-7733c464b507\") " Oct 31 19:20:32.232626 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.232577 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a063ed7-8ce0-414e-8bbf-7733c464b507-trusted-ca" pod "3a063ed7-8ce0-414e-8bbf-7733c464b507" (UID: "3a063ed7-8ce0-414e-8bbf-7733c464b507"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:20:32.232626 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.232604 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-certificates" pod "3a063ed7-8ce0-414e-8bbf-7733c464b507" (UID: "3a063ed7-8ce0-414e-8bbf-7733c464b507"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:20:32.234324 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.234275 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls" pod "3a063ed7-8ce0-414e-8bbf-7733c464b507" (UID: "3a063ed7-8ce0-414e-8bbf-7733c464b507"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:20:32.234433 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.234345 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a063ed7-8ce0-414e-8bbf-7733c464b507-installation-pull-secrets" pod "3a063ed7-8ce0-414e-8bbf-7733c464b507" (UID: "3a063ed7-8ce0-414e-8bbf-7733c464b507"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:20:32.234680 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.234643 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-kube-api-access-4zsbj" pod "3a063ed7-8ce0-414e-8bbf-7733c464b507" (UID: "3a063ed7-8ce0-414e-8bbf-7733c464b507"). InnerVolumeSpecName "kube-api-access-4zsbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:20:32.234788 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.234707 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-bound-sa-token" pod "3a063ed7-8ce0-414e-8bbf-7733c464b507" (UID: "3a063ed7-8ce0-414e-8bbf-7733c464b507"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:20:32.234853 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.234778 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a063ed7-8ce0-414e-8bbf-7733c464b507-image-registry-private-configuration" pod "3a063ed7-8ce0-414e-8bbf-7733c464b507" (UID: "3a063ed7-8ce0-414e-8bbf-7733c464b507"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:20:32.239176 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.239153 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a063ed7-8ce0-414e-8bbf-7733c464b507-ca-trust-extracted" pod "3a063ed7-8ce0-414e-8bbf-7733c464b507" (UID: "3a063ed7-8ce0-414e-8bbf-7733c464b507"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:20:32.332990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.332956 4910 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3a063ed7-8ce0-414e-8bbf-7733c464b507-installation-pull-secrets\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:20:32.332990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.332985 4910 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-bound-sa-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:20:32.332990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.332998 4910 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3a063ed7-8ce0-414e-8bbf-7733c464b507-image-registry-private-configuration\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:20:32.333219 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.333007 4910 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3a063ed7-8ce0-414e-8bbf-7733c464b507-ca-trust-extracted\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:20:32.333219 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.333017 4910 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-tls\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:20:32.333219 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.333025 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zsbj\" (UniqueName: \"kubernetes.io/projected/3a063ed7-8ce0-414e-8bbf-7733c464b507-kube-api-access-4zsbj\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:20:32.333219 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.333034 4910 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3a063ed7-8ce0-414e-8bbf-7733c464b507-trusted-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:20:32.333219 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.333044 4910 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3a063ed7-8ce0-414e-8bbf-7733c464b507-registry-certificates\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:20:32.428979 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.428945 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5c8ccd4bc6-xlml9"] Oct 31 19:20:32.434948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:32.434900 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5c8ccd4bc6-xlml9"] Oct 31 19:20:33.752002 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:33.751969 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a063ed7-8ce0-414e-8bbf-7733c464b507" path="/var/lib/kubelet/pods/3a063ed7-8ce0-414e-8bbf-7733c464b507/volumes" Oct 31 19:20:42.711352 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:20:42.711317 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:20:50.169920 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:50.169831 4910 generic.go:356] "Generic (PLEG): container finished" podID="ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f" containerID="cc31eee786c6c1ce374504d7227aa2d41018f11549d58a3d0a78202dbb3da80e" exitCode=0 Oct 31 19:20:50.169920 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:50.169891 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm" event={"ID":"ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f","Type":"ContainerDied","Data":"cc31eee786c6c1ce374504d7227aa2d41018f11549d58a3d0a78202dbb3da80e"} Oct 31 19:20:50.170345 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:50.170199 4910 scope.go:122] "RemoveContainer" containerID="cc31eee786c6c1ce374504d7227aa2d41018f11549d58a3d0a78202dbb3da80e" Oct 31 19:20:51.176108 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:51.176074 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-58hlm" event={"ID":"ae8a8fb6-c472-4e79-90bb-8a96c61e3b2f","Type":"ContainerStarted","Data":"1eb16d25c924bf7c523f75d6a4e5de9ce22e9646936c4a25bc66df60a95b1add"} Oct 31 19:20:55.191327 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:55.191249 4910 generic.go:356] "Generic (PLEG): container finished" podID="4ce261ba-08ff-418a-b3b4-b7e25e701bd8" containerID="d2d75ee097b7130386bb3b641ea0ee1129eb5cf4404817e71fec2f1596fc2c45" exitCode=0 Oct 31 19:20:55.191774 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:55.191322 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n" event={"ID":"4ce261ba-08ff-418a-b3b4-b7e25e701bd8","Type":"ContainerDied","Data":"d2d75ee097b7130386bb3b641ea0ee1129eb5cf4404817e71fec2f1596fc2c45"} Oct 31 19:20:55.191774 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:55.191605 4910 scope.go:122] "RemoveContainer" containerID="d2d75ee097b7130386bb3b641ea0ee1129eb5cf4404817e71fec2f1596fc2c45" Oct 31 19:20:56.196537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:20:56.196498 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-kts6n" event={"ID":"4ce261ba-08ff-418a-b3b4-b7e25e701bd8","Type":"ContainerStarted","Data":"4b23eba2dc0d8312922525d4b97d3734e1af4f31fb6917634c980c418d925051"} Oct 31 19:21:00.213121 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:21:00.213079 4910 generic.go:356] "Generic (PLEG): container finished" podID="599fc0e5-05d8-46b0-90b3-60818a371161" containerID="fde1d7db146568648f3672e51dd2f91b660f40070603b600b78a777d07445d44" exitCode=0 Oct 31 19:21:00.213502 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:21:00.213130 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mtmp6" event={"ID":"599fc0e5-05d8-46b0-90b3-60818a371161","Type":"ContainerDied","Data":"fde1d7db146568648f3672e51dd2f91b660f40070603b600b78a777d07445d44"} Oct 31 19:21:00.213502 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:21:00.213434 4910 scope.go:122] "RemoveContainer" containerID="fde1d7db146568648f3672e51dd2f91b660f40070603b600b78a777d07445d44" Oct 31 19:21:01.218582 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:21:01.218549 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mtmp6" event={"ID":"599fc0e5-05d8-46b0-90b3-60818a371161","Type":"ContainerStarted","Data":"f3e632d4042c1c50e8729d48f2a2865589d934684d34954b251b537205ce891e"} Oct 31 19:21:12.711588 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:21:12.711550 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:21:42.712129 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:21:42.712029 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:22:12.711101 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:22:12.711059 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:22:42.711082 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:22:42.711045 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:23:12.711553 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:23:12.711462 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:23:13.423111 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:13.423079 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t"] Oct 31 19:23:13.423373 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:13.423361 4910 cpu_manager.go:414] "containerMap: removing container" podUID="3a063ed7-8ce0-414e-8bbf-7733c464b507" containerName="registry" Oct 31 19:23:13.423417 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:13.423375 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a063ed7-8ce0-414e-8bbf-7733c464b507" containerName="registry" Oct 31 19:23:13.423417 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:13.423392 4910 cpu_manager.go:414] "containerMap: removing container" podUID="eb1a9bea-399b-4281-9ada-a232790cc713" containerName="registry" Oct 31 19:23:13.423417 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:13.423397 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb1a9bea-399b-4281-9ada-a232790cc713" containerName="registry" Oct 31 19:23:13.423509 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:13.423457 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a063ed7-8ce0-414e-8bbf-7733c464b507" containerName="registry" Oct 31 19:23:13.423509 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:13.423467 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb1a9bea-399b-4281-9ada-a232790cc713" containerName="registry" Oct 31 19:23:13.426514 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:13.426498 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t" Oct 31 19:23:13.438605 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:13.438578 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t"] Oct 31 19:23:13.448294 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:13.448273 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-j4zhp\"" Oct 31 19:23:13.449715 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:13.449694 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 19:23:13.449835 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:13.449716 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 19:23:13.538967 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:13.538901 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16802753-e91d-429f-b892-903715c63c97-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t\" (UID: \"16802753-e91d-429f-b892-903715c63c97\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t" Oct 31 19:23:13.538967 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:13.538969 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mndxf\" (UniqueName: \"kubernetes.io/projected/16802753-e91d-429f-b892-903715c63c97-kube-api-access-mndxf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t\" (UID: \"16802753-e91d-429f-b892-903715c63c97\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t" Oct 31 19:23:13.539197 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:13.539065 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16802753-e91d-429f-b892-903715c63c97-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t\" (UID: \"16802753-e91d-429f-b892-903715c63c97\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t" Oct 31 19:23:13.640214 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:13.640175 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16802753-e91d-429f-b892-903715c63c97-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t\" (UID: \"16802753-e91d-429f-b892-903715c63c97\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t" Oct 31 19:23:13.640409 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:13.640238 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16802753-e91d-429f-b892-903715c63c97-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t\" (UID: \"16802753-e91d-429f-b892-903715c63c97\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t" Oct 31 19:23:13.640409 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:13.640257 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mndxf\" (UniqueName: \"kubernetes.io/projected/16802753-e91d-429f-b892-903715c63c97-kube-api-access-mndxf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t\" (UID: \"16802753-e91d-429f-b892-903715c63c97\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t" Oct 31 19:23:13.640665 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:13.640638 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16802753-e91d-429f-b892-903715c63c97-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t\" (UID: \"16802753-e91d-429f-b892-903715c63c97\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t" Oct 31 19:23:13.640722 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:13.640647 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16802753-e91d-429f-b892-903715c63c97-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t\" (UID: \"16802753-e91d-429f-b892-903715c63c97\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t" Oct 31 19:23:13.656156 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:13.656116 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mndxf\" (UniqueName: \"kubernetes.io/projected/16802753-e91d-429f-b892-903715c63c97-kube-api-access-mndxf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t\" (UID: \"16802753-e91d-429f-b892-903715c63c97\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t" Oct 31 19:23:13.742108 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:13.742020 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t" Oct 31 19:23:14.014653 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:14.014547 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t"] Oct 31 19:23:14.017443 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:23:14.017415 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16802753_e91d_429f_b892_903715c63c97.slice/crio-384c4e28acb7af36a0ffb159362178bf7f2f53b212db9b18787a5dcf17400550 WatchSource:0}: Error finding container 384c4e28acb7af36a0ffb159362178bf7f2f53b212db9b18787a5dcf17400550: Status 404 returned error can't find the container with id 384c4e28acb7af36a0ffb159362178bf7f2f53b212db9b18787a5dcf17400550 Oct 31 19:23:14.708629 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:14.708588 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t" event={"ID":"16802753-e91d-429f-b892-903715c63c97","Type":"ContainerStarted","Data":"384c4e28acb7af36a0ffb159362178bf7f2f53b212db9b18787a5dcf17400550"} Oct 31 19:23:15.043531 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.043439 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z"] Oct 31 19:23:15.046990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.046966 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z" Oct 31 19:23:15.052543 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.052517 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-bqqzv\"" Oct 31 19:23:15.056688 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.056539 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z"] Oct 31 19:23:15.151028 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.150992 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc6zw\" (UniqueName: \"kubernetes.io/projected/019f8aee-81b0-479e-a666-49c24df1a5f4-kube-api-access-wc6zw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z\" (UID: \"019f8aee-81b0-479e-a666-49c24df1a5f4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z" Oct 31 19:23:15.151212 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.151050 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/019f8aee-81b0-479e-a666-49c24df1a5f4-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z\" (UID: \"019f8aee-81b0-479e-a666-49c24df1a5f4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z" Oct 31 19:23:15.151212 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.151143 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/019f8aee-81b0-479e-a666-49c24df1a5f4-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z\" (UID: \"019f8aee-81b0-479e-a666-49c24df1a5f4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z" Oct 31 19:23:15.251745 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.251705 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/019f8aee-81b0-479e-a666-49c24df1a5f4-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z\" (UID: \"019f8aee-81b0-479e-a666-49c24df1a5f4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z" Oct 31 19:23:15.251957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.251756 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/019f8aee-81b0-479e-a666-49c24df1a5f4-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z\" (UID: \"019f8aee-81b0-479e-a666-49c24df1a5f4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z" Oct 31 19:23:15.251957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.251836 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc6zw\" (UniqueName: \"kubernetes.io/projected/019f8aee-81b0-479e-a666-49c24df1a5f4-kube-api-access-wc6zw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z\" (UID: \"019f8aee-81b0-479e-a666-49c24df1a5f4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z" Oct 31 19:23:15.252283 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.252258 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/019f8aee-81b0-479e-a666-49c24df1a5f4-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z\" (UID: \"019f8aee-81b0-479e-a666-49c24df1a5f4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z" Oct 31 19:23:15.252341 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.252267 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/019f8aee-81b0-479e-a666-49c24df1a5f4-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z\" (UID: \"019f8aee-81b0-479e-a666-49c24df1a5f4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z" Oct 31 19:23:15.265621 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.265590 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc6zw\" (UniqueName: \"kubernetes.io/projected/019f8aee-81b0-479e-a666-49c24df1a5f4-kube-api-access-wc6zw\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z\" (UID: \"019f8aee-81b0-479e-a666-49c24df1a5f4\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z" Oct 31 19:23:15.367184 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.367146 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z" Oct 31 19:23:15.401694 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.400322 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2"] Oct 31 19:23:15.404069 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.404040 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2" Oct 31 19:23:15.409261 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.409218 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-r4kvv\"" Oct 31 19:23:15.414137 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.414062 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2"] Oct 31 19:23:15.506517 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.506485 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p"] Oct 31 19:23:15.510202 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.510179 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p" Oct 31 19:23:15.515427 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.515400 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-d7rm5\"" Oct 31 19:23:15.520943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.520901 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p"] Oct 31 19:23:15.554532 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.554501 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09afed6b-26e8-4a74-a6fc-20007942fc63-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2\" (UID: \"09afed6b-26e8-4a74-a6fc-20007942fc63\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2" Oct 31 19:23:15.554715 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.554573 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6q8m\" (UniqueName: \"kubernetes.io/projected/09afed6b-26e8-4a74-a6fc-20007942fc63-kube-api-access-m6q8m\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2\" (UID: \"09afed6b-26e8-4a74-a6fc-20007942fc63\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2" Oct 31 19:23:15.554715 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.554688 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09afed6b-26e8-4a74-a6fc-20007942fc63-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2\" (UID: \"09afed6b-26e8-4a74-a6fc-20007942fc63\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2" Oct 31 19:23:15.601265 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.601230 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq"] Oct 31 19:23:15.604958 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.604933 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq" Oct 31 19:23:15.608806 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.608784 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-f8cxm\"" Oct 31 19:23:15.615232 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.615041 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq"] Oct 31 19:23:15.648489 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.648416 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z"] Oct 31 19:23:15.655245 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.655217 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dc0fcfa0-2703-4241-87c9-1c7a45658853-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p\" (UID: \"dc0fcfa0-2703-4241-87c9-1c7a45658853\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p" Oct 31 19:23:15.655374 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.655261 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dc0fcfa0-2703-4241-87c9-1c7a45658853-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p\" (UID: \"dc0fcfa0-2703-4241-87c9-1c7a45658853\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p" Oct 31 19:23:15.655374 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.655307 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6q8m\" (UniqueName: \"kubernetes.io/projected/09afed6b-26e8-4a74-a6fc-20007942fc63-kube-api-access-m6q8m\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2\" (UID: \"09afed6b-26e8-4a74-a6fc-20007942fc63\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2" Oct 31 19:23:15.655487 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.655371 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09afed6b-26e8-4a74-a6fc-20007942fc63-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2\" (UID: \"09afed6b-26e8-4a74-a6fc-20007942fc63\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2" Oct 31 19:23:15.655487 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.655399 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kr5w\" (UniqueName: \"kubernetes.io/projected/dc0fcfa0-2703-4241-87c9-1c7a45658853-kube-api-access-9kr5w\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p\" (UID: \"dc0fcfa0-2703-4241-87c9-1c7a45658853\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p" Oct 31 19:23:15.655487 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.655429 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09afed6b-26e8-4a74-a6fc-20007942fc63-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2\" (UID: \"09afed6b-26e8-4a74-a6fc-20007942fc63\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2" Oct 31 19:23:15.655919 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.655881 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09afed6b-26e8-4a74-a6fc-20007942fc63-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2\" (UID: \"09afed6b-26e8-4a74-a6fc-20007942fc63\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2" Oct 31 19:23:15.656495 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:23:15.656468 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019f8aee_81b0_479e_a666_49c24df1a5f4.slice/crio-3f62bc2a21cb35e9502a6c69c2f565b4fc7af0170988d9980bbee9e9730b19ee WatchSource:0}: Error finding container 3f62bc2a21cb35e9502a6c69c2f565b4fc7af0170988d9980bbee9e9730b19ee: Status 404 returned error can't find the container with id 3f62bc2a21cb35e9502a6c69c2f565b4fc7af0170988d9980bbee9e9730b19ee Oct 31 19:23:15.656598 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.656505 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09afed6b-26e8-4a74-a6fc-20007942fc63-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2\" (UID: \"09afed6b-26e8-4a74-a6fc-20007942fc63\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2" Oct 31 19:23:15.669253 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.669226 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6q8m\" (UniqueName: \"kubernetes.io/projected/09afed6b-26e8-4a74-a6fc-20007942fc63-kube-api-access-m6q8m\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2\" (UID: \"09afed6b-26e8-4a74-a6fc-20007942fc63\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2" Oct 31 19:23:15.702256 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.702225 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn"] Oct 31 19:23:15.707734 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.707708 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn" Oct 31 19:23:15.714319 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.714268 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-9k7j6\"" Oct 31 19:23:15.714319 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.714307 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z" event={"ID":"019f8aee-81b0-479e-a666-49c24df1a5f4","Type":"ContainerStarted","Data":"3f62bc2a21cb35e9502a6c69c2f565b4fc7af0170988d9980bbee9e9730b19ee"} Oct 31 19:23:15.716019 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.715997 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn"] Oct 31 19:23:15.732165 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.732141 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2" Oct 31 19:23:15.756352 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.756323 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kr5w\" (UniqueName: \"kubernetes.io/projected/dc0fcfa0-2703-4241-87c9-1c7a45658853-kube-api-access-9kr5w\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p\" (UID: \"dc0fcfa0-2703-4241-87c9-1c7a45658853\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p" Oct 31 19:23:15.756483 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.756369 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqppz\" (UniqueName: \"kubernetes.io/projected/46e644cf-c804-4672-86bd-bd8207a5c8c8-kube-api-access-cqppz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq\" (UID: \"46e644cf-c804-4672-86bd-bd8207a5c8c8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq" Oct 31 19:23:15.756483 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.756411 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dc0fcfa0-2703-4241-87c9-1c7a45658853-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p\" (UID: \"dc0fcfa0-2703-4241-87c9-1c7a45658853\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p" Oct 31 19:23:15.756483 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.756431 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dc0fcfa0-2703-4241-87c9-1c7a45658853-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p\" (UID: \"dc0fcfa0-2703-4241-87c9-1c7a45658853\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p" Oct 31 19:23:15.756483 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.756456 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46e644cf-c804-4672-86bd-bd8207a5c8c8-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq\" (UID: \"46e644cf-c804-4672-86bd-bd8207a5c8c8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq" Oct 31 19:23:15.756645 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.756494 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46e644cf-c804-4672-86bd-bd8207a5c8c8-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq\" (UID: \"46e644cf-c804-4672-86bd-bd8207a5c8c8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq" Oct 31 19:23:15.756846 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.756827 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dc0fcfa0-2703-4241-87c9-1c7a45658853-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p\" (UID: \"dc0fcfa0-2703-4241-87c9-1c7a45658853\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p" Oct 31 19:23:15.756978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.756960 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dc0fcfa0-2703-4241-87c9-1c7a45658853-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p\" (UID: \"dc0fcfa0-2703-4241-87c9-1c7a45658853\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p" Oct 31 19:23:15.769659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.769627 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kr5w\" (UniqueName: \"kubernetes.io/projected/dc0fcfa0-2703-4241-87c9-1c7a45658853-kube-api-access-9kr5w\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p\" (UID: \"dc0fcfa0-2703-4241-87c9-1c7a45658853\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p" Oct 31 19:23:15.829366 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.829331 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p" Oct 31 19:23:15.857906 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.857865 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46e644cf-c804-4672-86bd-bd8207a5c8c8-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq\" (UID: \"46e644cf-c804-4672-86bd-bd8207a5c8c8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq" Oct 31 19:23:15.858062 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.858047 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46e644cf-c804-4672-86bd-bd8207a5c8c8-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq\" (UID: \"46e644cf-c804-4672-86bd-bd8207a5c8c8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq" Oct 31 19:23:15.858182 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.858092 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqppz\" (UniqueName: \"kubernetes.io/projected/46e644cf-c804-4672-86bd-bd8207a5c8c8-kube-api-access-cqppz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq\" (UID: \"46e644cf-c804-4672-86bd-bd8207a5c8c8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq" Oct 31 19:23:15.858182 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.858125 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/940880ae-0658-443d-b641-cc8371d797c7-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn\" (UID: \"940880ae-0658-443d-b641-cc8371d797c7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn" Oct 31 19:23:15.858182 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.858154 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrc7v\" (UniqueName: \"kubernetes.io/projected/940880ae-0658-443d-b641-cc8371d797c7-kube-api-access-zrc7v\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn\" (UID: \"940880ae-0658-443d-b641-cc8371d797c7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn" Oct 31 19:23:15.858349 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.858211 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/940880ae-0658-443d-b641-cc8371d797c7-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn\" (UID: \"940880ae-0658-443d-b641-cc8371d797c7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn" Oct 31 19:23:15.858349 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.858276 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46e644cf-c804-4672-86bd-bd8207a5c8c8-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq\" (UID: \"46e644cf-c804-4672-86bd-bd8207a5c8c8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq" Oct 31 19:23:15.858349 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.858343 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46e644cf-c804-4672-86bd-bd8207a5c8c8-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq\" (UID: \"46e644cf-c804-4672-86bd-bd8207a5c8c8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq" Oct 31 19:23:15.873016 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.872989 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqppz\" (UniqueName: \"kubernetes.io/projected/46e644cf-c804-4672-86bd-bd8207a5c8c8-kube-api-access-cqppz\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq\" (UID: \"46e644cf-c804-4672-86bd-bd8207a5c8c8\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq" Oct 31 19:23:15.924604 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.924528 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq" Oct 31 19:23:15.958856 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.958821 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/940880ae-0658-443d-b641-cc8371d797c7-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn\" (UID: \"940880ae-0658-443d-b641-cc8371d797c7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn" Oct 31 19:23:15.959049 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.958959 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrc7v\" (UniqueName: \"kubernetes.io/projected/940880ae-0658-443d-b641-cc8371d797c7-kube-api-access-zrc7v\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn\" (UID: \"940880ae-0658-443d-b641-cc8371d797c7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn" Oct 31 19:23:15.959049 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.959021 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/940880ae-0658-443d-b641-cc8371d797c7-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn\" (UID: \"940880ae-0658-443d-b641-cc8371d797c7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn" Oct 31 19:23:15.959317 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.959289 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/940880ae-0658-443d-b641-cc8371d797c7-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn\" (UID: \"940880ae-0658-443d-b641-cc8371d797c7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn" Oct 31 19:23:15.959444 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.959413 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/940880ae-0658-443d-b641-cc8371d797c7-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn\" (UID: \"940880ae-0658-443d-b641-cc8371d797c7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn" Oct 31 19:23:15.973056 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:15.972952 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrc7v\" (UniqueName: \"kubernetes.io/projected/940880ae-0658-443d-b641-cc8371d797c7-kube-api-access-zrc7v\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn\" (UID: \"940880ae-0658-443d-b641-cc8371d797c7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn" Oct 31 19:23:16.017242 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:16.017209 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2"] Oct 31 19:23:16.020844 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:23:16.020813 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09afed6b_26e8_4a74_a6fc_20007942fc63.slice/crio-54ba532588368c35d493c3c40a9842a69f69c02fa38303e72c5b8633f7d0702d WatchSource:0}: Error finding container 54ba532588368c35d493c3c40a9842a69f69c02fa38303e72c5b8633f7d0702d: Status 404 returned error can't find the container with id 54ba532588368c35d493c3c40a9842a69f69c02fa38303e72c5b8633f7d0702d Oct 31 19:23:16.027342 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:16.027316 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn" Oct 31 19:23:16.115469 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:16.115416 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p"] Oct 31 19:23:16.117802 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:23:16.117774 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc0fcfa0_2703_4241_87c9_1c7a45658853.slice/crio-37ab8a95a49533892c4bff43f5acf622eae5ca1da533ad2f95f86b44806f3350 WatchSource:0}: Error finding container 37ab8a95a49533892c4bff43f5acf622eae5ca1da533ad2f95f86b44806f3350: Status 404 returned error can't find the container with id 37ab8a95a49533892c4bff43f5acf622eae5ca1da533ad2f95f86b44806f3350 Oct 31 19:23:16.212734 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:16.212666 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq"] Oct 31 19:23:16.216266 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:23:16.216235 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e644cf_c804_4672_86bd_bd8207a5c8c8.slice/crio-b8fca5dd25e273867ca2d5e8f097b84f6099912f502a9623d6010548546199c9 WatchSource:0}: Error finding container b8fca5dd25e273867ca2d5e8f097b84f6099912f502a9623d6010548546199c9: Status 404 returned error can't find the container with id b8fca5dd25e273867ca2d5e8f097b84f6099912f502a9623d6010548546199c9 Oct 31 19:23:16.319243 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:16.319212 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn"] Oct 31 19:23:16.322367 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:23:16.322330 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940880ae_0658_443d_b641_cc8371d797c7.slice/crio-6d8c063cb1f4122ad15f4aa80f3dbebeb2161b0c719aec5db962c447338254e1 WatchSource:0}: Error finding container 6d8c063cb1f4122ad15f4aa80f3dbebeb2161b0c719aec5db962c447338254e1: Status 404 returned error can't find the container with id 6d8c063cb1f4122ad15f4aa80f3dbebeb2161b0c719aec5db962c447338254e1 Oct 31 19:23:16.725030 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:16.724981 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p" event={"ID":"dc0fcfa0-2703-4241-87c9-1c7a45658853","Type":"ContainerStarted","Data":"37ab8a95a49533892c4bff43f5acf622eae5ca1da533ad2f95f86b44806f3350"} Oct 31 19:23:16.728071 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:16.727975 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2" event={"ID":"09afed6b-26e8-4a74-a6fc-20007942fc63","Type":"ContainerStarted","Data":"54ba532588368c35d493c3c40a9842a69f69c02fa38303e72c5b8633f7d0702d"} Oct 31 19:23:16.729769 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:16.729710 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn" event={"ID":"940880ae-0658-443d-b641-cc8371d797c7","Type":"ContainerStarted","Data":"6d8c063cb1f4122ad15f4aa80f3dbebeb2161b0c719aec5db962c447338254e1"} Oct 31 19:23:16.731295 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:16.731244 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq" event={"ID":"46e644cf-c804-4672-86bd-bd8207a5c8c8","Type":"ContainerStarted","Data":"b8fca5dd25e273867ca2d5e8f097b84f6099912f502a9623d6010548546199c9"} Oct 31 19:23:18.527129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.527094 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h"] Oct 31 19:23:18.530728 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.530702 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.546939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.543105 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn"] Oct 31 19:23:18.548098 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.547298 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 19:23:18.548098 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.548012 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-jkgv9\"" Oct 31 19:23:18.550929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.550521 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h"] Oct 31 19:23:18.550929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.550636 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn" Oct 31 19:23:18.558763 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.558734 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn"] Oct 31 19:23:18.594378 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.594348 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-hgwh2\"" Oct 31 19:23:18.659458 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.659427 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79"] Oct 31 19:23:18.663593 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.663567 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.676580 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.676526 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-dtqc8\"" Oct 31 19:23:18.684401 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.684374 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b43015-708e-4ef2-883e-cc121fe1c8e5-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.684531 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.684429 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b43015-708e-4ef2-883e-cc121fe1c8e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.684531 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.684475 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b43015-708e-4ef2-883e-cc121fe1c8e5-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.684531 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.684503 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bshr8\" (UniqueName: \"kubernetes.io/projected/f5b43015-708e-4ef2-883e-cc121fe1c8e5-kube-api-access-bshr8\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.684531 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.684528 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b43015-708e-4ef2-883e-cc121fe1c8e5-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.684755 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.684579 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b43015-708e-4ef2-883e-cc121fe1c8e5-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.684755 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.684606 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b43015-708e-4ef2-883e-cc121fe1c8e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.684755 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.684690 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b43015-708e-4ef2-883e-cc121fe1c8e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.684755 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.684716 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d717eda-76de-491d-9bb0-bd375c10dd02-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn\" (UID: \"1d717eda-76de-491d-9bb0-bd375c10dd02\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn" Oct 31 19:23:18.684755 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.684752 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b43015-708e-4ef2-883e-cc121fe1c8e5-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.685035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.684776 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d717eda-76de-491d-9bb0-bd375c10dd02-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn\" (UID: \"1d717eda-76de-491d-9bb0-bd375c10dd02\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn" Oct 31 19:23:18.685035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.684819 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfhrz\" (UniqueName: \"kubernetes.io/projected/1d717eda-76de-491d-9bb0-bd375c10dd02-kube-api-access-bfhrz\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn\" (UID: \"1d717eda-76de-491d-9bb0-bd375c10dd02\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn" Oct 31 19:23:18.685930 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.685894 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79"] Oct 31 19:23:18.785992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.785385 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b43015-708e-4ef2-883e-cc121fe1c8e5-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.785992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.785428 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d717eda-76de-491d-9bb0-bd375c10dd02-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn\" (UID: \"1d717eda-76de-491d-9bb0-bd375c10dd02\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn" Oct 31 19:23:18.785992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.785456 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfhrz\" (UniqueName: \"kubernetes.io/projected/1d717eda-76de-491d-9bb0-bd375c10dd02-kube-api-access-bfhrz\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn\" (UID: \"1d717eda-76de-491d-9bb0-bd375c10dd02\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn" Oct 31 19:23:18.785992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.785488 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3f9166-9fbc-49b5-8292-90096c2f6c25-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.785992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.785519 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b43015-708e-4ef2-883e-cc121fe1c8e5-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.785992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.785583 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b43015-708e-4ef2-883e-cc121fe1c8e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.785992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.785613 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d3f9166-9fbc-49b5-8292-90096c2f6c25-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.785992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.785655 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b43015-708e-4ef2-883e-cc121fe1c8e5-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.785992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.785683 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bshr8\" (UniqueName: \"kubernetes.io/projected/f5b43015-708e-4ef2-883e-cc121fe1c8e5-kube-api-access-bshr8\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.785992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.785711 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b43015-708e-4ef2-883e-cc121fe1c8e5-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.785992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.785749 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d3f9166-9fbc-49b5-8292-90096c2f6c25-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.785992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.785778 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3f9166-9fbc-49b5-8292-90096c2f6c25-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.785992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.785808 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b43015-708e-4ef2-883e-cc121fe1c8e5-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.785992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.785832 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b43015-708e-4ef2-883e-cc121fe1c8e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.785992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.785864 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d3f9166-9fbc-49b5-8292-90096c2f6c25-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.785992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.785933 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d3f9166-9fbc-49b5-8292-90096c2f6c25-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.785992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.785970 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6ctv\" (UniqueName: \"kubernetes.io/projected/8d3f9166-9fbc-49b5-8292-90096c2f6c25-kube-api-access-z6ctv\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.785992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.786001 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d3f9166-9fbc-49b5-8292-90096c2f6c25-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.786733 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.786045 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b43015-708e-4ef2-883e-cc121fe1c8e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.786733 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.786072 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d717eda-76de-491d-9bb0-bd375c10dd02-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn\" (UID: \"1d717eda-76de-491d-9bb0-bd375c10dd02\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn" Oct 31 19:23:18.786733 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.786097 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d3f9166-9fbc-49b5-8292-90096c2f6c25-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.786733 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.786591 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b43015-708e-4ef2-883e-cc121fe1c8e5-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.786923 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.786886 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d717eda-76de-491d-9bb0-bd375c10dd02-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn\" (UID: \"1d717eda-76de-491d-9bb0-bd375c10dd02\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn" Oct 31 19:23:18.788503 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.787973 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b43015-708e-4ef2-883e-cc121fe1c8e5-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.788503 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.788169 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b43015-708e-4ef2-883e-cc121fe1c8e5-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.788503 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.788382 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b43015-708e-4ef2-883e-cc121fe1c8e5-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.789476 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.789153 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b43015-708e-4ef2-883e-cc121fe1c8e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.789476 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.789360 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d717eda-76de-491d-9bb0-bd375c10dd02-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn\" (UID: \"1d717eda-76de-491d-9bb0-bd375c10dd02\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn" Oct 31 19:23:18.798156 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.798131 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b43015-708e-4ef2-883e-cc121fe1c8e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.816723 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.816691 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b43015-708e-4ef2-883e-cc121fe1c8e5-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.818433 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.818408 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b43015-708e-4ef2-883e-cc121fe1c8e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.818943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.818898 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bshr8\" (UniqueName: \"kubernetes.io/projected/f5b43015-708e-4ef2-883e-cc121fe1c8e5-kube-api-access-bshr8\") pod \"maas-default-gateway-openshift-default-75d7c785f-w7w5h\" (UID: \"f5b43015-708e-4ef2-883e-cc121fe1c8e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.820182 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.820158 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfhrz\" (UniqueName: \"kubernetes.io/projected/1d717eda-76de-491d-9bb0-bd375c10dd02-kube-api-access-bfhrz\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn\" (UID: \"1d717eda-76de-491d-9bb0-bd375c10dd02\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn" Oct 31 19:23:18.856261 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.856225 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:18.875656 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.875627 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn" Oct 31 19:23:18.887929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.887239 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d3f9166-9fbc-49b5-8292-90096c2f6c25-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.887929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.887285 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6ctv\" (UniqueName: \"kubernetes.io/projected/8d3f9166-9fbc-49b5-8292-90096c2f6c25-kube-api-access-z6ctv\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.887929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.887319 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d3f9166-9fbc-49b5-8292-90096c2f6c25-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.887929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.887373 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d3f9166-9fbc-49b5-8292-90096c2f6c25-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.887929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.887423 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3f9166-9fbc-49b5-8292-90096c2f6c25-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.887929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.887485 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d3f9166-9fbc-49b5-8292-90096c2f6c25-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.887929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.887530 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d3f9166-9fbc-49b5-8292-90096c2f6c25-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.887929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.887557 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3f9166-9fbc-49b5-8292-90096c2f6c25-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.887929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.887592 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d3f9166-9fbc-49b5-8292-90096c2f6c25-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.888518 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.888309 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d3f9166-9fbc-49b5-8292-90096c2f6c25-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.888573 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.888538 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d3f9166-9fbc-49b5-8292-90096c2f6c25-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.890808 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.890151 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d3f9166-9fbc-49b5-8292-90096c2f6c25-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.890808 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.890393 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3f9166-9fbc-49b5-8292-90096c2f6c25-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.890808 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.890504 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d3f9166-9fbc-49b5-8292-90096c2f6c25-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.890808 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.890743 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3f9166-9fbc-49b5-8292-90096c2f6c25-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.896982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.896401 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d3f9166-9fbc-49b5-8292-90096c2f6c25-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.927147 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.927079 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6ctv\" (UniqueName: \"kubernetes.io/projected/8d3f9166-9fbc-49b5-8292-90096c2f6c25-kube-api-access-z6ctv\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.928973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.928781 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d3f9166-9fbc-49b5-8292-90096c2f6c25-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79\" (UID: \"8d3f9166-9fbc-49b5-8292-90096c2f6c25\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:18.982505 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:18.982216 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:19.198986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:19.198954 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn"] Oct 31 19:23:19.202986 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:23:19.202944 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d717eda_76de_491d_9bb0_bd375c10dd02.slice/crio-444272349fd33dd7892293846d8e85302197318282e9ecc2248e298678a64da4 WatchSource:0}: Error finding container 444272349fd33dd7892293846d8e85302197318282e9ecc2248e298678a64da4: Status 404 returned error can't find the container with id 444272349fd33dd7892293846d8e85302197318282e9ecc2248e298678a64da4 Oct 31 19:23:19.264176 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:19.263660 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h"] Oct 31 19:23:19.265550 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:23:19.265497 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b43015_708e_4ef2_883e_cc121fe1c8e5.slice/crio-d817e0b107aba3e0440ffb3dcc6e0ef07d0e7539ecc1fd2efd49cad3015f10ed WatchSource:0}: Error finding container d817e0b107aba3e0440ffb3dcc6e0ef07d0e7539ecc1fd2efd49cad3015f10ed: Status 404 returned error can't find the container with id d817e0b107aba3e0440ffb3dcc6e0ef07d0e7539ecc1fd2efd49cad3015f10ed Oct 31 19:23:19.336386 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:19.336267 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79"] Oct 31 19:23:19.775199 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:19.775111 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn" event={"ID":"1d717eda-76de-491d-9bb0-bd375c10dd02","Type":"ContainerStarted","Data":"444272349fd33dd7892293846d8e85302197318282e9ecc2248e298678a64da4"} Oct 31 19:23:19.778969 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:19.778935 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" event={"ID":"f5b43015-708e-4ef2-883e-cc121fe1c8e5","Type":"ContainerStarted","Data":"d817e0b107aba3e0440ffb3dcc6e0ef07d0e7539ecc1fd2efd49cad3015f10ed"} Oct 31 19:23:19.781131 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:19.781108 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" event={"ID":"8d3f9166-9fbc-49b5-8292-90096c2f6c25","Type":"ContainerStarted","Data":"af4d6ba713056c511a6167bf67fedfbc4e15472d0cead95ef03923c1a218424a"} Oct 31 19:23:27.875441 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:27.875398 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:23:27.875844 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:27.875479 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:23:27.875844 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:27.875524 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:23:27.895492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:27.895459 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:23:27.895584 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:27.895555 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:23:27.895655 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:27.895629 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:23:28.906657 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:28.906484 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" event={"ID":"8d3f9166-9fbc-49b5-8292-90096c2f6c25","Type":"ContainerStarted","Data":"d795d33f475d02e0dd4c4d73db08f200bb3eccd2d78dcae2a632f492bcd033b1"} Oct 31 19:23:28.911577 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:28.911524 4910 generic.go:356] "Generic (PLEG): container finished" podID="16802753-e91d-429f-b892-903715c63c97" containerID="fc062889d5c282aff433097fcedcff3d6cdf833471b57ac70e17a4f321407fc6" exitCode=0 Oct 31 19:23:28.911706 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:28.911604 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t" event={"ID":"16802753-e91d-429f-b892-903715c63c97","Type":"ContainerDied","Data":"fc062889d5c282aff433097fcedcff3d6cdf833471b57ac70e17a4f321407fc6"} Oct 31 19:23:28.915545 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:28.915418 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" event={"ID":"f5b43015-708e-4ef2-883e-cc121fe1c8e5","Type":"ContainerStarted","Data":"03bde39d6987c97f3e6d8b5a285dfff0a6cfc48bcfd979afabbbb2d535ae0bd9"} Oct 31 19:23:28.937170 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:28.937110 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" podStartSLOduration=2.416331739 podStartE2EDuration="10.937089316s" totalImagesPullingTime="8.520757577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:23:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:23:28.934228276 +0000 UTC m=+243.297048676" watchObservedRunningTime="2026-10-31 19:23:28.937089316 +0000 UTC m=+243.299909710" Oct 31 19:23:28.982640 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:28.982557 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:29.857178 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:29.857148 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:29.862994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:29.862711 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:29.893334 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:29.892060 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" podStartSLOduration=3.26789826 podStartE2EDuration="11.89203981s" totalImagesPullingTime="8.62414155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:23:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:23:28.982980133 +0000 UTC m=+243.345800506" watchObservedRunningTime="2026-10-31 19:23:29.89203981 +0000 UTC m=+244.254860184" Oct 31 19:23:29.923525 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:29.923494 4910 generic.go:356] "Generic (PLEG): container finished" podID="1d717eda-76de-491d-9bb0-bd375c10dd02" containerID="d4d26e284c32bcd43fbb3dd113b0fd8c2b5b6f18b853020a2e37eb11f30da876" exitCode=0 Oct 31 19:23:29.924784 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:29.924763 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn" event={"ID":"1d717eda-76de-491d-9bb0-bd375c10dd02","Type":"ContainerDied","Data":"d4d26e284c32bcd43fbb3dd113b0fd8c2b5b6f18b853020a2e37eb11f30da876"} Oct 31 19:23:29.924839 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:29.924808 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:29.926071 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:29.925994 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:23:29.990532 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:29.990505 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:30.935736 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:30.935690 4910 generic.go:356] "Generic (PLEG): container finished" podID="019f8aee-81b0-479e-a666-49c24df1a5f4" containerID="bb670837fe27431edab5bf48b1053073b84f130506d7503481c7fda4a0525e84" exitCode=0 Oct 31 19:23:30.936234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:30.935750 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z" event={"ID":"019f8aee-81b0-479e-a666-49c24df1a5f4","Type":"ContainerDied","Data":"bb670837fe27431edab5bf48b1053073b84f130506d7503481c7fda4a0525e84"} Oct 31 19:23:30.936532 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:30.936513 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:30.939215 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:30.939196 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79" Oct 31 19:23:31.942652 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:31.942615 4910 generic.go:356] "Generic (PLEG): container finished" podID="09afed6b-26e8-4a74-a6fc-20007942fc63" containerID="7dd78fcdad441b98c43900c15fd697698c7f246ed6e64f4578538cb2b599649e" exitCode=0 Oct 31 19:23:31.943169 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:31.942709 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2" event={"ID":"09afed6b-26e8-4a74-a6fc-20007942fc63","Type":"ContainerDied","Data":"7dd78fcdad441b98c43900c15fd697698c7f246ed6e64f4578538cb2b599649e"} Oct 31 19:23:31.945558 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:31.945047 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn" event={"ID":"940880ae-0658-443d-b641-cc8371d797c7","Type":"ContainerStarted","Data":"19341624dfb17c7f09f70540fe824aa60cc7d629f0f7a3994bc44a89ca8e63eb"} Oct 31 19:23:31.946891 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:31.946851 4910 generic.go:356] "Generic (PLEG): container finished" podID="dc0fcfa0-2703-4241-87c9-1c7a45658853" containerID="761658da054568fb0be3ce15d19afff9a2d0f5a9b60a7c558e692d9269328c31" exitCode=0 Oct 31 19:23:31.947023 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:31.946943 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p" event={"ID":"dc0fcfa0-2703-4241-87c9-1c7a45658853","Type":"ContainerDied","Data":"761658da054568fb0be3ce15d19afff9a2d0f5a9b60a7c558e692d9269328c31"} Oct 31 19:23:32.953893 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:32.953815 4910 generic.go:356] "Generic (PLEG): container finished" podID="019f8aee-81b0-479e-a666-49c24df1a5f4" containerID="8b2de2707eb36a1d3b009c3e175bd14a3e7320836a8fc657b482fdaeb64b5e2f" exitCode=0 Oct 31 19:23:32.954362 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:32.953900 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z" event={"ID":"019f8aee-81b0-479e-a666-49c24df1a5f4","Type":"ContainerDied","Data":"8b2de2707eb36a1d3b009c3e175bd14a3e7320836a8fc657b482fdaeb64b5e2f"} Oct 31 19:23:32.956129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:32.956098 4910 generic.go:356] "Generic (PLEG): container finished" podID="940880ae-0658-443d-b641-cc8371d797c7" containerID="19341624dfb17c7f09f70540fe824aa60cc7d629f0f7a3994bc44a89ca8e63eb" exitCode=0 Oct 31 19:23:32.956270 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:32.956190 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn" event={"ID":"940880ae-0658-443d-b641-cc8371d797c7","Type":"ContainerDied","Data":"19341624dfb17c7f09f70540fe824aa60cc7d629f0f7a3994bc44a89ca8e63eb"} Oct 31 19:23:32.958125 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:32.958100 4910 generic.go:356] "Generic (PLEG): container finished" podID="46e644cf-c804-4672-86bd-bd8207a5c8c8" containerID="01ca9faf05c1555209ab1849d98d99492cdbab5a9aae015ab1a3625b500ec262" exitCode=0 Oct 31 19:23:32.958243 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:32.958136 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq" event={"ID":"46e644cf-c804-4672-86bd-bd8207a5c8c8","Type":"ContainerDied","Data":"01ca9faf05c1555209ab1849d98d99492cdbab5a9aae015ab1a3625b500ec262"} Oct 31 19:23:32.960262 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:32.960241 4910 generic.go:356] "Generic (PLEG): container finished" podID="16802753-e91d-429f-b892-903715c63c97" containerID="b1183dc43405f2c5753be50697fb88dfc0a7f255c4c403238a23fe45f780c4d0" exitCode=0 Oct 31 19:23:32.960352 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:32.960322 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t" event={"ID":"16802753-e91d-429f-b892-903715c63c97","Type":"ContainerDied","Data":"b1183dc43405f2c5753be50697fb88dfc0a7f255c4c403238a23fe45f780c4d0"} Oct 31 19:23:33.969851 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:33.969731 4910 generic.go:356] "Generic (PLEG): container finished" podID="dc0fcfa0-2703-4241-87c9-1c7a45658853" containerID="b695fd4f7ac46a62d5e23d2eb26fc49535a49453fe77bd275b36121da0bc9243" exitCode=0 Oct 31 19:23:33.970464 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:33.969844 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p" event={"ID":"dc0fcfa0-2703-4241-87c9-1c7a45658853","Type":"ContainerDied","Data":"b695fd4f7ac46a62d5e23d2eb26fc49535a49453fe77bd275b36121da0bc9243"} Oct 31 19:23:33.973237 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:33.972588 4910 generic.go:356] "Generic (PLEG): container finished" podID="1d717eda-76de-491d-9bb0-bd375c10dd02" containerID="23c065625d1f954ca8581d5f4589b49cd6214f722e47c5f4504c5cb2f66e66c2" exitCode=0 Oct 31 19:23:33.973237 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:33.972729 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn" event={"ID":"1d717eda-76de-491d-9bb0-bd375c10dd02","Type":"ContainerDied","Data":"23c065625d1f954ca8581d5f4589b49cd6214f722e47c5f4504c5cb2f66e66c2"} Oct 31 19:23:33.975522 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:33.975373 4910 generic.go:356] "Generic (PLEG): container finished" podID="09afed6b-26e8-4a74-a6fc-20007942fc63" containerID="0dd9ff3d95dc4d1ff11e6c8816e1998837e7515663c736f3b76c203f7c870622" exitCode=0 Oct 31 19:23:33.975522 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:33.975436 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2" event={"ID":"09afed6b-26e8-4a74-a6fc-20007942fc63","Type":"ContainerDied","Data":"0dd9ff3d95dc4d1ff11e6c8816e1998837e7515663c736f3b76c203f7c870622"} Oct 31 19:23:34.985965 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:34.985631 4910 generic.go:356] "Generic (PLEG): container finished" podID="46e644cf-c804-4672-86bd-bd8207a5c8c8" containerID="3ef40a65f0305efc8c5d1d4f2a7b77621cd97c399f37af69f91059d1b4091cd9" exitCode=0 Oct 31 19:23:34.985965 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:34.985724 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq" event={"ID":"46e644cf-c804-4672-86bd-bd8207a5c8c8","Type":"ContainerDied","Data":"3ef40a65f0305efc8c5d1d4f2a7b77621cd97c399f37af69f91059d1b4091cd9"} Oct 31 19:23:34.989051 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:34.989026 4910 generic.go:356] "Generic (PLEG): container finished" podID="940880ae-0658-443d-b641-cc8371d797c7" containerID="274c38d957e4043c06aa3edcc87762a9a92eb14e953102b9bde933eaa8a00832" exitCode=0 Oct 31 19:23:34.989172 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:34.989061 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn" event={"ID":"940880ae-0658-443d-b641-cc8371d797c7","Type":"ContainerDied","Data":"274c38d957e4043c06aa3edcc87762a9a92eb14e953102b9bde933eaa8a00832"} Oct 31 19:23:42.752004 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:23:42.751889 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:23:47.101332 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:47.101295 4910 generic.go:356] "Generic (PLEG): container finished" podID="16802753-e91d-429f-b892-903715c63c97" containerID="36e5041683ac4e83d4dfc38cc67fcbc555aad041132349c70a50a8ba192f4f7c" exitCode=0 Oct 31 19:23:47.101799 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:47.101429 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t" event={"ID":"16802753-e91d-429f-b892-903715c63c97","Type":"ContainerDied","Data":"36e5041683ac4e83d4dfc38cc67fcbc555aad041132349c70a50a8ba192f4f7c"} Oct 31 19:23:48.110133 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:48.110091 4910 generic.go:356] "Generic (PLEG): container finished" podID="019f8aee-81b0-479e-a666-49c24df1a5f4" containerID="91eb712b56be4aa8a92bc7d65d968cdb952d9fb114d2d7b165f757375cab11e9" exitCode=0 Oct 31 19:23:48.110710 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:48.110404 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z" event={"ID":"019f8aee-81b0-479e-a666-49c24df1a5f4","Type":"ContainerDied","Data":"91eb712b56be4aa8a92bc7d65d968cdb952d9fb114d2d7b165f757375cab11e9"} Oct 31 19:23:48.278682 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:48.278345 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t" Oct 31 19:23:48.392853 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:48.392827 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16802753-e91d-429f-b892-903715c63c97-kube-api-access-mndxf\" (UniqueName: \"kubernetes.io/projected/16802753-e91d-429f-b892-903715c63c97-kube-api-access-mndxf\") pod \"16802753-e91d-429f-b892-903715c63c97\" (UID: \"16802753-e91d-429f-b892-903715c63c97\") " Oct 31 19:23:48.393000 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:48.392887 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16802753-e91d-429f-b892-903715c63c97-util\" (UniqueName: \"kubernetes.io/empty-dir/16802753-e91d-429f-b892-903715c63c97-util\") pod \"16802753-e91d-429f-b892-903715c63c97\" (UID: \"16802753-e91d-429f-b892-903715c63c97\") " Oct 31 19:23:48.393000 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:48.392962 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16802753-e91d-429f-b892-903715c63c97-bundle\" (UniqueName: \"kubernetes.io/empty-dir/16802753-e91d-429f-b892-903715c63c97-bundle\") pod \"16802753-e91d-429f-b892-903715c63c97\" (UID: \"16802753-e91d-429f-b892-903715c63c97\") " Oct 31 19:23:48.393963 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:48.393933 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16802753-e91d-429f-b892-903715c63c97-bundle" pod "16802753-e91d-429f-b892-903715c63c97" (UID: "16802753-e91d-429f-b892-903715c63c97"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:23:48.395732 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:48.395705 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16802753-e91d-429f-b892-903715c63c97-kube-api-access-mndxf" pod "16802753-e91d-429f-b892-903715c63c97" (UID: "16802753-e91d-429f-b892-903715c63c97"). InnerVolumeSpecName "kube-api-access-mndxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:23:48.399715 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:48.399689 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16802753-e91d-429f-b892-903715c63c97-util" pod "16802753-e91d-429f-b892-903715c63c97" (UID: "16802753-e91d-429f-b892-903715c63c97"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:23:48.493879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:48.493840 4910 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/16802753-e91d-429f-b892-903715c63c97-bundle\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:48.493879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:48.493881 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mndxf\" (UniqueName: \"kubernetes.io/projected/16802753-e91d-429f-b892-903715c63c97-kube-api-access-mndxf\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:48.493879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:48.493892 4910 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/16802753-e91d-429f-b892-903715c63c97-util\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:49.119629 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:49.119591 4910 generic.go:356] "Generic (PLEG): container finished" podID="09afed6b-26e8-4a74-a6fc-20007942fc63" containerID="cc7a41999657e6cfb5b8a141d737ee107349b8c1a094b2e22b3bb3c8b4b455b2" exitCode=0 Oct 31 19:23:49.120101 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:49.119683 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2" event={"ID":"09afed6b-26e8-4a74-a6fc-20007942fc63","Type":"ContainerDied","Data":"cc7a41999657e6cfb5b8a141d737ee107349b8c1a094b2e22b3bb3c8b4b455b2"} Oct 31 19:23:49.122606 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:49.122568 4910 generic.go:356] "Generic (PLEG): container finished" podID="dc0fcfa0-2703-4241-87c9-1c7a45658853" containerID="f072cf8ea2ace0c021def9f49552097e4eb332fe0f8d3462fb494801e94bd1bb" exitCode=0 Oct 31 19:23:49.122752 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:49.122671 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p" event={"ID":"dc0fcfa0-2703-4241-87c9-1c7a45658853","Type":"ContainerDied","Data":"f072cf8ea2ace0c021def9f49552097e4eb332fe0f8d3462fb494801e94bd1bb"} Oct 31 19:23:49.125475 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:49.125447 4910 generic.go:356] "Generic (PLEG): container finished" podID="1d717eda-76de-491d-9bb0-bd375c10dd02" containerID="005370c3f8190e6879b1a0a939faa6e42d36195f2d3f6b5a41c685830a373eef" exitCode=0 Oct 31 19:23:49.125608 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:49.125562 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn" event={"ID":"1d717eda-76de-491d-9bb0-bd375c10dd02","Type":"ContainerDied","Data":"005370c3f8190e6879b1a0a939faa6e42d36195f2d3f6b5a41c685830a373eef"} Oct 31 19:23:49.127831 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:49.127812 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t" Oct 31 19:23:49.127948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:49.127844 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cgqm5t" event={"ID":"16802753-e91d-429f-b892-903715c63c97","Type":"ContainerDied","Data":"384c4e28acb7af36a0ffb159362178bf7f2f53b212db9b18787a5dcf17400550"} Oct 31 19:23:49.127948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:49.127886 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="384c4e28acb7af36a0ffb159362178bf7f2f53b212db9b18787a5dcf17400550" Oct 31 19:23:49.315035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:49.315005 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z" Oct 31 19:23:49.401790 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:49.401751 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/019f8aee-81b0-479e-a666-49c24df1a5f4-util\" (UniqueName: \"kubernetes.io/empty-dir/019f8aee-81b0-479e-a666-49c24df1a5f4-util\") pod \"019f8aee-81b0-479e-a666-49c24df1a5f4\" (UID: \"019f8aee-81b0-479e-a666-49c24df1a5f4\") " Oct 31 19:23:49.401985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:49.401844 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/019f8aee-81b0-479e-a666-49c24df1a5f4-kube-api-access-wc6zw\" (UniqueName: \"kubernetes.io/projected/019f8aee-81b0-479e-a666-49c24df1a5f4-kube-api-access-wc6zw\") pod \"019f8aee-81b0-479e-a666-49c24df1a5f4\" (UID: \"019f8aee-81b0-479e-a666-49c24df1a5f4\") " Oct 31 19:23:49.402032 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:49.401986 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/019f8aee-81b0-479e-a666-49c24df1a5f4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/019f8aee-81b0-479e-a666-49c24df1a5f4-bundle\") pod \"019f8aee-81b0-479e-a666-49c24df1a5f4\" (UID: \"019f8aee-81b0-479e-a666-49c24df1a5f4\") " Oct 31 19:23:49.402516 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:49.402487 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/019f8aee-81b0-479e-a666-49c24df1a5f4-bundle" pod "019f8aee-81b0-479e-a666-49c24df1a5f4" (UID: "019f8aee-81b0-479e-a666-49c24df1a5f4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:23:49.404003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:49.403969 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/019f8aee-81b0-479e-a666-49c24df1a5f4-kube-api-access-wc6zw" pod "019f8aee-81b0-479e-a666-49c24df1a5f4" (UID: "019f8aee-81b0-479e-a666-49c24df1a5f4"). InnerVolumeSpecName "kube-api-access-wc6zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:23:49.406978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:49.406956 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/019f8aee-81b0-479e-a666-49c24df1a5f4-util" pod "019f8aee-81b0-479e-a666-49c24df1a5f4" (UID: "019f8aee-81b0-479e-a666-49c24df1a5f4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:23:49.503549 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:49.503481 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc6zw\" (UniqueName: \"kubernetes.io/projected/019f8aee-81b0-479e-a666-49c24df1a5f4-kube-api-access-wc6zw\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:49.503549 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:49.503514 4910 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/019f8aee-81b0-479e-a666-49c24df1a5f4-bundle\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:49.503549 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:49.503524 4910 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/019f8aee-81b0-479e-a666-49c24df1a5f4-util\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:50.134227 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.134193 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z" event={"ID":"019f8aee-81b0-479e-a666-49c24df1a5f4","Type":"ContainerDied","Data":"3f62bc2a21cb35e9502a6c69c2f565b4fc7af0170988d9980bbee9e9730b19ee"} Oct 31 19:23:50.134678 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.134244 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f62bc2a21cb35e9502a6c69c2f565b4fc7af0170988d9980bbee9e9730b19ee" Oct 31 19:23:50.134678 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.134206 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xpn5z" Oct 31 19:23:50.136655 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.136535 4910 generic.go:356] "Generic (PLEG): container finished" podID="940880ae-0658-443d-b641-cc8371d797c7" containerID="a55a180cd55c3b4954e079f255ade60e31a1b442d283062559652ca93462a275" exitCode=0 Oct 31 19:23:50.136655 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.136634 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn" event={"ID":"940880ae-0658-443d-b641-cc8371d797c7","Type":"ContainerDied","Data":"a55a180cd55c3b4954e079f255ade60e31a1b442d283062559652ca93462a275"} Oct 31 19:23:50.138729 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.138710 4910 generic.go:356] "Generic (PLEG): container finished" podID="46e644cf-c804-4672-86bd-bd8207a5c8c8" containerID="052f32c42c733100a4774db5e80c33feb1b33c1b41417cc56a587bb493f3a769" exitCode=0 Oct 31 19:23:50.138816 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.138732 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq" event={"ID":"46e644cf-c804-4672-86bd-bd8207a5c8c8","Type":"ContainerDied","Data":"052f32c42c733100a4774db5e80c33feb1b33c1b41417cc56a587bb493f3a769"} Oct 31 19:23:50.309102 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.309075 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p" Oct 31 19:23:50.349217 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.349193 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn" Oct 31 19:23:50.350530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.350514 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2" Oct 31 19:23:50.427702 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.427668 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc0fcfa0-2703-4241-87c9-1c7a45658853-util\" (UniqueName: \"kubernetes.io/empty-dir/dc0fcfa0-2703-4241-87c9-1c7a45658853-util\") pod \"dc0fcfa0-2703-4241-87c9-1c7a45658853\" (UID: \"dc0fcfa0-2703-4241-87c9-1c7a45658853\") " Oct 31 19:23:50.427858 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.427723 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc0fcfa0-2703-4241-87c9-1c7a45658853-kube-api-access-9kr5w\" (UniqueName: \"kubernetes.io/projected/dc0fcfa0-2703-4241-87c9-1c7a45658853-kube-api-access-9kr5w\") pod \"dc0fcfa0-2703-4241-87c9-1c7a45658853\" (UID: \"dc0fcfa0-2703-4241-87c9-1c7a45658853\") " Oct 31 19:23:50.427858 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.427784 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc0fcfa0-2703-4241-87c9-1c7a45658853-bundle\" (UniqueName: \"kubernetes.io/empty-dir/dc0fcfa0-2703-4241-87c9-1c7a45658853-bundle\") pod \"dc0fcfa0-2703-4241-87c9-1c7a45658853\" (UID: \"dc0fcfa0-2703-4241-87c9-1c7a45658853\") " Oct 31 19:23:50.428345 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.428309 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc0fcfa0-2703-4241-87c9-1c7a45658853-bundle" pod "dc0fcfa0-2703-4241-87c9-1c7a45658853" (UID: "dc0fcfa0-2703-4241-87c9-1c7a45658853"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:23:50.429940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.429901 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc0fcfa0-2703-4241-87c9-1c7a45658853-kube-api-access-9kr5w" pod "dc0fcfa0-2703-4241-87c9-1c7a45658853" (UID: "dc0fcfa0-2703-4241-87c9-1c7a45658853"). InnerVolumeSpecName "kube-api-access-9kr5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:23:50.433246 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.433226 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc0fcfa0-2703-4241-87c9-1c7a45658853-util" pod "dc0fcfa0-2703-4241-87c9-1c7a45658853" (UID: "dc0fcfa0-2703-4241-87c9-1c7a45658853"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:23:50.528503 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.528457 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09afed6b-26e8-4a74-a6fc-20007942fc63-util\" (UniqueName: \"kubernetes.io/empty-dir/09afed6b-26e8-4a74-a6fc-20007942fc63-util\") pod \"09afed6b-26e8-4a74-a6fc-20007942fc63\" (UID: \"09afed6b-26e8-4a74-a6fc-20007942fc63\") " Oct 31 19:23:50.528705 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.528523 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09afed6b-26e8-4a74-a6fc-20007942fc63-kube-api-access-m6q8m\" (UniqueName: \"kubernetes.io/projected/09afed6b-26e8-4a74-a6fc-20007942fc63-kube-api-access-m6q8m\") pod \"09afed6b-26e8-4a74-a6fc-20007942fc63\" (UID: \"09afed6b-26e8-4a74-a6fc-20007942fc63\") " Oct 31 19:23:50.528705 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.528575 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d717eda-76de-491d-9bb0-bd375c10dd02-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d717eda-76de-491d-9bb0-bd375c10dd02-bundle\") pod \"1d717eda-76de-491d-9bb0-bd375c10dd02\" (UID: \"1d717eda-76de-491d-9bb0-bd375c10dd02\") " Oct 31 19:23:50.528705 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.528602 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09afed6b-26e8-4a74-a6fc-20007942fc63-bundle\" (UniqueName: \"kubernetes.io/empty-dir/09afed6b-26e8-4a74-a6fc-20007942fc63-bundle\") pod \"09afed6b-26e8-4a74-a6fc-20007942fc63\" (UID: \"09afed6b-26e8-4a74-a6fc-20007942fc63\") " Oct 31 19:23:50.528705 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.528643 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d717eda-76de-491d-9bb0-bd375c10dd02-util\" (UniqueName: \"kubernetes.io/empty-dir/1d717eda-76de-491d-9bb0-bd375c10dd02-util\") pod \"1d717eda-76de-491d-9bb0-bd375c10dd02\" (UID: \"1d717eda-76de-491d-9bb0-bd375c10dd02\") " Oct 31 19:23:50.528705 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.528668 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d717eda-76de-491d-9bb0-bd375c10dd02-kube-api-access-bfhrz\" (UniqueName: \"kubernetes.io/projected/1d717eda-76de-491d-9bb0-bd375c10dd02-kube-api-access-bfhrz\") pod \"1d717eda-76de-491d-9bb0-bd375c10dd02\" (UID: \"1d717eda-76de-491d-9bb0-bd375c10dd02\") " Oct 31 19:23:50.529012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.528896 4910 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dc0fcfa0-2703-4241-87c9-1c7a45658853-util\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:50.529012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.528944 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kr5w\" (UniqueName: \"kubernetes.io/projected/dc0fcfa0-2703-4241-87c9-1c7a45658853-kube-api-access-9kr5w\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:50.529012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.528960 4910 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dc0fcfa0-2703-4241-87c9-1c7a45658853-bundle\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:50.529156 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.529031 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d717eda-76de-491d-9bb0-bd375c10dd02-bundle" pod "1d717eda-76de-491d-9bb0-bd375c10dd02" (UID: "1d717eda-76de-491d-9bb0-bd375c10dd02"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:23:50.529221 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.529179 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09afed6b-26e8-4a74-a6fc-20007942fc63-bundle" pod "09afed6b-26e8-4a74-a6fc-20007942fc63" (UID: "09afed6b-26e8-4a74-a6fc-20007942fc63"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:23:50.530716 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.530688 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09afed6b-26e8-4a74-a6fc-20007942fc63-kube-api-access-m6q8m" pod "09afed6b-26e8-4a74-a6fc-20007942fc63" (UID: "09afed6b-26e8-4a74-a6fc-20007942fc63"). InnerVolumeSpecName "kube-api-access-m6q8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:23:50.530819 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.530727 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d717eda-76de-491d-9bb0-bd375c10dd02-kube-api-access-bfhrz" pod "1d717eda-76de-491d-9bb0-bd375c10dd02" (UID: "1d717eda-76de-491d-9bb0-bd375c10dd02"). InnerVolumeSpecName "kube-api-access-bfhrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:23:50.534677 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.534654 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d717eda-76de-491d-9bb0-bd375c10dd02-util" pod "1d717eda-76de-491d-9bb0-bd375c10dd02" (UID: "1d717eda-76de-491d-9bb0-bd375c10dd02"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:23:50.534892 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.534868 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09afed6b-26e8-4a74-a6fc-20007942fc63-util" pod "09afed6b-26e8-4a74-a6fc-20007942fc63" (UID: "09afed6b-26e8-4a74-a6fc-20007942fc63"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:23:50.629529 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.629429 4910 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09afed6b-26e8-4a74-a6fc-20007942fc63-util\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:50.629529 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.629472 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6q8m\" (UniqueName: \"kubernetes.io/projected/09afed6b-26e8-4a74-a6fc-20007942fc63-kube-api-access-m6q8m\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:50.629529 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.629486 4910 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d717eda-76de-491d-9bb0-bd375c10dd02-bundle\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:50.629529 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.629500 4910 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09afed6b-26e8-4a74-a6fc-20007942fc63-bundle\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:50.629529 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.629510 4910 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d717eda-76de-491d-9bb0-bd375c10dd02-util\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:50.629529 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:50.629521 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfhrz\" (UniqueName: \"kubernetes.io/projected/1d717eda-76de-491d-9bb0-bd375c10dd02-kube-api-access-bfhrz\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:51.145412 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.145380 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2" Oct 31 19:23:51.145986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.145411 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckbvv2" event={"ID":"09afed6b-26e8-4a74-a6fc-20007942fc63","Type":"ContainerDied","Data":"54ba532588368c35d493c3c40a9842a69f69c02fa38303e72c5b8633f7d0702d"} Oct 31 19:23:51.145986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.145470 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54ba532588368c35d493c3c40a9842a69f69c02fa38303e72c5b8633f7d0702d" Oct 31 19:23:51.147451 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.147422 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p" event={"ID":"dc0fcfa0-2703-4241-87c9-1c7a45658853","Type":"ContainerDied","Data":"37ab8a95a49533892c4bff43f5acf622eae5ca1da533ad2f95f86b44806f3350"} Oct 31 19:23:51.147575 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.147468 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37ab8a95a49533892c4bff43f5acf622eae5ca1da533ad2f95f86b44806f3350" Oct 31 19:23:51.147575 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.147427 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7mjk2p" Oct 31 19:23:51.149312 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.149286 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn" event={"ID":"1d717eda-76de-491d-9bb0-bd375c10dd02","Type":"ContainerDied","Data":"444272349fd33dd7892293846d8e85302197318282e9ecc2248e298678a64da4"} Oct 31 19:23:51.149425 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.149327 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="444272349fd33dd7892293846d8e85302197318282e9ecc2248e298678a64da4" Oct 31 19:23:51.149425 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.149352 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644q5sjn" Oct 31 19:23:51.280387 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.280361 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq" Oct 31 19:23:51.308710 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.308688 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn" Oct 31 19:23:51.437073 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.436995 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940880ae-0658-443d-b641-cc8371d797c7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/940880ae-0658-443d-b641-cc8371d797c7-bundle\") pod \"940880ae-0658-443d-b641-cc8371d797c7\" (UID: \"940880ae-0658-443d-b641-cc8371d797c7\") " Oct 31 19:23:51.437073 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.437039 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/940880ae-0658-443d-b641-cc8371d797c7-kube-api-access-zrc7v\" (UniqueName: \"kubernetes.io/projected/940880ae-0658-443d-b641-cc8371d797c7-kube-api-access-zrc7v\") pod \"940880ae-0658-443d-b641-cc8371d797c7\" (UID: \"940880ae-0658-443d-b641-cc8371d797c7\") " Oct 31 19:23:51.437299 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.437093 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46e644cf-c804-4672-86bd-bd8207a5c8c8-kube-api-access-cqppz\" (UniqueName: \"kubernetes.io/projected/46e644cf-c804-4672-86bd-bd8207a5c8c8-kube-api-access-cqppz\") pod \"46e644cf-c804-4672-86bd-bd8207a5c8c8\" (UID: \"46e644cf-c804-4672-86bd-bd8207a5c8c8\") " Oct 31 19:23:51.437299 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.437127 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940880ae-0658-443d-b641-cc8371d797c7-util\" (UniqueName: \"kubernetes.io/empty-dir/940880ae-0658-443d-b641-cc8371d797c7-util\") pod \"940880ae-0658-443d-b641-cc8371d797c7\" (UID: \"940880ae-0658-443d-b641-cc8371d797c7\") " Oct 31 19:23:51.437299 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.437174 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e644cf-c804-4672-86bd-bd8207a5c8c8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/46e644cf-c804-4672-86bd-bd8207a5c8c8-bundle\") pod \"46e644cf-c804-4672-86bd-bd8207a5c8c8\" (UID: \"46e644cf-c804-4672-86bd-bd8207a5c8c8\") " Oct 31 19:23:51.437299 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.437230 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e644cf-c804-4672-86bd-bd8207a5c8c8-util\" (UniqueName: \"kubernetes.io/empty-dir/46e644cf-c804-4672-86bd-bd8207a5c8c8-util\") pod \"46e644cf-c804-4672-86bd-bd8207a5c8c8\" (UID: \"46e644cf-c804-4672-86bd-bd8207a5c8c8\") " Oct 31 19:23:51.437692 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.437660 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e644cf-c804-4672-86bd-bd8207a5c8c8-bundle" pod "46e644cf-c804-4672-86bd-bd8207a5c8c8" (UID: "46e644cf-c804-4672-86bd-bd8207a5c8c8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:23:51.437835 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.437805 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940880ae-0658-443d-b641-cc8371d797c7-bundle" pod "940880ae-0658-443d-b641-cc8371d797c7" (UID: "940880ae-0658-443d-b641-cc8371d797c7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:23:51.439257 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.439231 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e644cf-c804-4672-86bd-bd8207a5c8c8-kube-api-access-cqppz" pod "46e644cf-c804-4672-86bd-bd8207a5c8c8" (UID: "46e644cf-c804-4672-86bd-bd8207a5c8c8"). InnerVolumeSpecName "kube-api-access-cqppz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:23:51.440349 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.440319 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940880ae-0658-443d-b641-cc8371d797c7-kube-api-access-zrc7v" pod "940880ae-0658-443d-b641-cc8371d797c7" (UID: "940880ae-0658-443d-b641-cc8371d797c7"). InnerVolumeSpecName "kube-api-access-zrc7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:23:51.443109 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.443084 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940880ae-0658-443d-b641-cc8371d797c7-util" pod "940880ae-0658-443d-b641-cc8371d797c7" (UID: "940880ae-0658-443d-b641-cc8371d797c7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:23:51.443218 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.443196 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e644cf-c804-4672-86bd-bd8207a5c8c8-util" pod "46e644cf-c804-4672-86bd-bd8207a5c8c8" (UID: "46e644cf-c804-4672-86bd-bd8207a5c8c8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:23:51.538580 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.538544 4910 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46e644cf-c804-4672-86bd-bd8207a5c8c8-bundle\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:51.538580 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.538575 4910 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46e644cf-c804-4672-86bd-bd8207a5c8c8-util\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:51.538580 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.538584 4910 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/940880ae-0658-443d-b641-cc8371d797c7-bundle\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:51.538814 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.538593 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrc7v\" (UniqueName: \"kubernetes.io/projected/940880ae-0658-443d-b641-cc8371d797c7-kube-api-access-zrc7v\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:51.538814 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.538603 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqppz\" (UniqueName: \"kubernetes.io/projected/46e644cf-c804-4672-86bd-bd8207a5c8c8-kube-api-access-cqppz\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:51.538814 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:51.538611 4910 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/940880ae-0658-443d-b641-cc8371d797c7-util\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:23:52.158101 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:52.158070 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn" Oct 31 19:23:52.158101 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:52.158077 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847agj6dn" event={"ID":"940880ae-0658-443d-b641-cc8371d797c7","Type":"ContainerDied","Data":"6d8c063cb1f4122ad15f4aa80f3dbebeb2161b0c719aec5db962c447338254e1"} Oct 31 19:23:52.158599 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:52.158134 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d8c063cb1f4122ad15f4aa80f3dbebeb2161b0c719aec5db962c447338254e1" Oct 31 19:23:52.160373 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:52.160349 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq" event={"ID":"46e644cf-c804-4672-86bd-bd8207a5c8c8","Type":"ContainerDied","Data":"b8fca5dd25e273867ca2d5e8f097b84f6099912f502a9623d6010548546199c9"} Oct 31 19:23:52.160502 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:52.160390 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8fca5dd25e273867ca2d5e8f097b84f6099912f502a9623d6010548546199c9" Oct 31 19:23:52.160502 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:52.160394 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6f5sxq" Oct 31 19:23:56.639413 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639378 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm"] Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639680 4910 cpu_manager.go:414] "containerMap: removing container" podUID="46e644cf-c804-4672-86bd-bd8207a5c8c8" containerName="pull" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639690 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46e644cf-c804-4672-86bd-bd8207a5c8c8" containerName="pull" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639699 4910 cpu_manager.go:414] "containerMap: removing container" podUID="09afed6b-26e8-4a74-a6fc-20007942fc63" containerName="extract" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639704 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09afed6b-26e8-4a74-a6fc-20007942fc63" containerName="extract" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639712 4910 cpu_manager.go:414] "containerMap: removing container" podUID="1d717eda-76de-491d-9bb0-bd375c10dd02" containerName="extract" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639718 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d717eda-76de-491d-9bb0-bd375c10dd02" containerName="extract" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639726 4910 cpu_manager.go:414] "containerMap: removing container" podUID="940880ae-0658-443d-b641-cc8371d797c7" containerName="pull" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639731 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940880ae-0658-443d-b641-cc8371d797c7" containerName="pull" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639741 4910 cpu_manager.go:414] "containerMap: removing container" podUID="dc0fcfa0-2703-4241-87c9-1c7a45658853" containerName="pull" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639745 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc0fcfa0-2703-4241-87c9-1c7a45658853" containerName="pull" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639755 4910 cpu_manager.go:414] "containerMap: removing container" podUID="09afed6b-26e8-4a74-a6fc-20007942fc63" containerName="pull" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639760 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09afed6b-26e8-4a74-a6fc-20007942fc63" containerName="pull" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639766 4910 cpu_manager.go:414] "containerMap: removing container" podUID="019f8aee-81b0-479e-a666-49c24df1a5f4" containerName="extract" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639770 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="019f8aee-81b0-479e-a666-49c24df1a5f4" containerName="extract" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639778 4910 cpu_manager.go:414] "containerMap: removing container" podUID="1d717eda-76de-491d-9bb0-bd375c10dd02" containerName="util" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639782 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d717eda-76de-491d-9bb0-bd375c10dd02" containerName="util" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639790 4910 cpu_manager.go:414] "containerMap: removing container" podUID="46e644cf-c804-4672-86bd-bd8207a5c8c8" containerName="extract" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639795 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46e644cf-c804-4672-86bd-bd8207a5c8c8" containerName="extract" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639803 4910 cpu_manager.go:414] "containerMap: removing container" podUID="16802753-e91d-429f-b892-903715c63c97" containerName="util" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639807 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16802753-e91d-429f-b892-903715c63c97" containerName="util" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639813 4910 cpu_manager.go:414] "containerMap: removing container" podUID="09afed6b-26e8-4a74-a6fc-20007942fc63" containerName="util" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639817 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09afed6b-26e8-4a74-a6fc-20007942fc63" containerName="util" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639824 4910 cpu_manager.go:414] "containerMap: removing container" podUID="dc0fcfa0-2703-4241-87c9-1c7a45658853" containerName="extract" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639830 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc0fcfa0-2703-4241-87c9-1c7a45658853" containerName="extract" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639838 4910 cpu_manager.go:414] "containerMap: removing container" podUID="019f8aee-81b0-479e-a666-49c24df1a5f4" containerName="util" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639842 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="019f8aee-81b0-479e-a666-49c24df1a5f4" containerName="util" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639847 4910 cpu_manager.go:414] "containerMap: removing container" podUID="16802753-e91d-429f-b892-903715c63c97" containerName="extract" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639852 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16802753-e91d-429f-b892-903715c63c97" containerName="extract" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639859 4910 cpu_manager.go:414] "containerMap: removing container" podUID="940880ae-0658-443d-b641-cc8371d797c7" containerName="util" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639863 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940880ae-0658-443d-b641-cc8371d797c7" containerName="util" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639870 4910 cpu_manager.go:414] "containerMap: removing container" podUID="1d717eda-76de-491d-9bb0-bd375c10dd02" containerName="pull" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639875 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d717eda-76de-491d-9bb0-bd375c10dd02" containerName="pull" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639882 4910 cpu_manager.go:414] "containerMap: removing container" podUID="46e644cf-c804-4672-86bd-bd8207a5c8c8" containerName="util" Oct 31 19:23:56.639868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639886 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46e644cf-c804-4672-86bd-bd8207a5c8c8" containerName="util" Oct 31 19:23:56.640997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639891 4910 cpu_manager.go:414] "containerMap: removing container" podUID="019f8aee-81b0-479e-a666-49c24df1a5f4" containerName="pull" Oct 31 19:23:56.640997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639895 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="019f8aee-81b0-479e-a666-49c24df1a5f4" containerName="pull" Oct 31 19:23:56.640997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639902 4910 cpu_manager.go:414] "containerMap: removing container" podUID="16802753-e91d-429f-b892-903715c63c97" containerName="pull" Oct 31 19:23:56.640997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639906 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16802753-e91d-429f-b892-903715c63c97" containerName="pull" Oct 31 19:23:56.640997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639926 4910 cpu_manager.go:414] "containerMap: removing container" podUID="dc0fcfa0-2703-4241-87c9-1c7a45658853" containerName="util" Oct 31 19:23:56.640997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639930 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc0fcfa0-2703-4241-87c9-1c7a45658853" containerName="util" Oct 31 19:23:56.640997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639936 4910 cpu_manager.go:414] "containerMap: removing container" podUID="940880ae-0658-443d-b641-cc8371d797c7" containerName="extract" Oct 31 19:23:56.640997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639940 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940880ae-0658-443d-b641-cc8371d797c7" containerName="extract" Oct 31 19:23:56.640997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639987 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="019f8aee-81b0-479e-a666-49c24df1a5f4" containerName="extract" Oct 31 19:23:56.640997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.639994 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="46e644cf-c804-4672-86bd-bd8207a5c8c8" containerName="extract" Oct 31 19:23:56.640997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.640001 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc0fcfa0-2703-4241-87c9-1c7a45658853" containerName="extract" Oct 31 19:23:56.640997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.640008 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="16802753-e91d-429f-b892-903715c63c97" containerName="extract" Oct 31 19:23:56.640997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.640014 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="940880ae-0658-443d-b641-cc8371d797c7" containerName="extract" Oct 31 19:23:56.640997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.640019 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="09afed6b-26e8-4a74-a6fc-20007942fc63" containerName="extract" Oct 31 19:23:56.640997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.640024 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d717eda-76de-491d-9bb0-bd375c10dd02" containerName="extract" Oct 31 19:23:56.644308 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.644290 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" Oct 31 19:23:56.650156 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.650134 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 19:23:56.650752 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.650726 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 19:23:56.650752 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.650740 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 19:23:56.650959 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.650751 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 19:23:56.651266 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.650963 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-hzfp2\"" Oct 31 19:23:56.653229 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.653212 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 19:23:56.780384 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.780346 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e164cd97-6fd0-49c6-b20d-9f30bffaca59-manifests\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-ptzqm\" (UID: \"e164cd97-6fd0-49c6-b20d-9f30bffaca59\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" Oct 31 19:23:56.780543 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.780499 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e164cd97-6fd0-49c6-b20d-9f30bffaca59-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-ptzqm\" (UID: \"e164cd97-6fd0-49c6-b20d-9f30bffaca59\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" Oct 31 19:23:56.780604 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.780540 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e164cd97-6fd0-49c6-b20d-9f30bffaca59-tmp\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-ptzqm\" (UID: \"e164cd97-6fd0-49c6-b20d-9f30bffaca59\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" Oct 31 19:23:56.780648 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.780606 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e164cd97-6fd0-49c6-b20d-9f30bffaca59-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-ptzqm\" (UID: \"e164cd97-6fd0-49c6-b20d-9f30bffaca59\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" Oct 31 19:23:56.780688 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.780672 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqxwx\" (UniqueName: \"kubernetes.io/projected/e164cd97-6fd0-49c6-b20d-9f30bffaca59-kube-api-access-cqxwx\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-ptzqm\" (UID: \"e164cd97-6fd0-49c6-b20d-9f30bffaca59\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" Oct 31 19:23:56.780762 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.780734 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e164cd97-6fd0-49c6-b20d-9f30bffaca59-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-ptzqm\" (UID: \"e164cd97-6fd0-49c6-b20d-9f30bffaca59\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" Oct 31 19:23:56.782320 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.782299 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm"] Oct 31 19:23:56.882008 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.881979 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e164cd97-6fd0-49c6-b20d-9f30bffaca59-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-ptzqm\" (UID: \"e164cd97-6fd0-49c6-b20d-9f30bffaca59\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" Oct 31 19:23:56.882166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.882017 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e164cd97-6fd0-49c6-b20d-9f30bffaca59-tmp\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-ptzqm\" (UID: \"e164cd97-6fd0-49c6-b20d-9f30bffaca59\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" Oct 31 19:23:56.882166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.882036 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e164cd97-6fd0-49c6-b20d-9f30bffaca59-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-ptzqm\" (UID: \"e164cd97-6fd0-49c6-b20d-9f30bffaca59\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" Oct 31 19:23:56.882166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.882059 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqxwx\" (UniqueName: \"kubernetes.io/projected/e164cd97-6fd0-49c6-b20d-9f30bffaca59-kube-api-access-cqxwx\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-ptzqm\" (UID: \"e164cd97-6fd0-49c6-b20d-9f30bffaca59\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" Oct 31 19:23:56.882166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.882110 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e164cd97-6fd0-49c6-b20d-9f30bffaca59-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-ptzqm\" (UID: \"e164cd97-6fd0-49c6-b20d-9f30bffaca59\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" Oct 31 19:23:56.882166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.882135 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e164cd97-6fd0-49c6-b20d-9f30bffaca59-manifests\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-ptzqm\" (UID: \"e164cd97-6fd0-49c6-b20d-9f30bffaca59\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" Oct 31 19:23:56.882895 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.882678 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e164cd97-6fd0-49c6-b20d-9f30bffaca59-manifests\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-ptzqm\" (UID: \"e164cd97-6fd0-49c6-b20d-9f30bffaca59\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" Oct 31 19:23:56.884671 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.884646 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e164cd97-6fd0-49c6-b20d-9f30bffaca59-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-ptzqm\" (UID: \"e164cd97-6fd0-49c6-b20d-9f30bffaca59\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" Oct 31 19:23:56.884793 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.884692 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e164cd97-6fd0-49c6-b20d-9f30bffaca59-tmp\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-ptzqm\" (UID: \"e164cd97-6fd0-49c6-b20d-9f30bffaca59\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" Oct 31 19:23:56.884793 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.884781 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e164cd97-6fd0-49c6-b20d-9f30bffaca59-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-ptzqm\" (UID: \"e164cd97-6fd0-49c6-b20d-9f30bffaca59\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" Oct 31 19:23:56.884901 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.884846 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e164cd97-6fd0-49c6-b20d-9f30bffaca59-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-ptzqm\" (UID: \"e164cd97-6fd0-49c6-b20d-9f30bffaca59\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" Oct 31 19:23:56.903549 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.903505 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqxwx\" (UniqueName: \"kubernetes.io/projected/e164cd97-6fd0-49c6-b20d-9f30bffaca59-kube-api-access-cqxwx\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-ptzqm\" (UID: \"e164cd97-6fd0-49c6-b20d-9f30bffaca59\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" Oct 31 19:23:56.959278 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:56.959255 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" Oct 31 19:23:57.388706 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:23:57.388676 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode164cd97_6fd0_49c6_b20d_9f30bffaca59.slice/crio-63353de7cdc39d8e23a9ebcdac59b6d0f30e7e24124e5b429ede3eeba745a17b WatchSource:0}: Error finding container 63353de7cdc39d8e23a9ebcdac59b6d0f30e7e24124e5b429ede3eeba745a17b: Status 404 returned error can't find the container with id 63353de7cdc39d8e23a9ebcdac59b6d0f30e7e24124e5b429ede3eeba745a17b Oct 31 19:23:57.389179 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:57.389152 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm"] Oct 31 19:23:57.782153 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:57.782076 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-pdmbp"] Oct 31 19:23:57.786708 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:57.786687 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-pdmbp" Oct 31 19:23:57.801526 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:57.801507 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 19:23:57.801613 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:57.801600 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 19:23:57.806670 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:57.806650 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-pdmbp"] Oct 31 19:23:57.810250 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:57.810231 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-89mzm\"" Oct 31 19:23:57.891644 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:57.891614 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c0853112-0260-4a67-93d3-b885b7944089-bound-sa-token\") pod \"cert-manager-597b95cb84-pdmbp\" (UID: \"c0853112-0260-4a67-93d3-b885b7944089\") " pod="cert-manager/cert-manager-597b95cb84-pdmbp" Oct 31 19:23:57.891772 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:57.891661 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt5nz\" (UniqueName: \"kubernetes.io/projected/c0853112-0260-4a67-93d3-b885b7944089-kube-api-access-gt5nz\") pod \"cert-manager-597b95cb84-pdmbp\" (UID: \"c0853112-0260-4a67-93d3-b885b7944089\") " pod="cert-manager/cert-manager-597b95cb84-pdmbp" Oct 31 19:23:57.992882 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:57.992849 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c0853112-0260-4a67-93d3-b885b7944089-bound-sa-token\") pod \"cert-manager-597b95cb84-pdmbp\" (UID: \"c0853112-0260-4a67-93d3-b885b7944089\") " pod="cert-manager/cert-manager-597b95cb84-pdmbp" Oct 31 19:23:57.996986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:57.993297 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt5nz\" (UniqueName: \"kubernetes.io/projected/c0853112-0260-4a67-93d3-b885b7944089-kube-api-access-gt5nz\") pod \"cert-manager-597b95cb84-pdmbp\" (UID: \"c0853112-0260-4a67-93d3-b885b7944089\") " pod="cert-manager/cert-manager-597b95cb84-pdmbp" Oct 31 19:23:58.008536 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:58.008508 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c0853112-0260-4a67-93d3-b885b7944089-bound-sa-token\") pod \"cert-manager-597b95cb84-pdmbp\" (UID: \"c0853112-0260-4a67-93d3-b885b7944089\") " pod="cert-manager/cert-manager-597b95cb84-pdmbp" Oct 31 19:23:58.009162 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:58.009139 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt5nz\" (UniqueName: \"kubernetes.io/projected/c0853112-0260-4a67-93d3-b885b7944089-kube-api-access-gt5nz\") pod \"cert-manager-597b95cb84-pdmbp\" (UID: \"c0853112-0260-4a67-93d3-b885b7944089\") " pod="cert-manager/cert-manager-597b95cb84-pdmbp" Oct 31 19:23:58.108118 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:58.108070 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-pdmbp" Oct 31 19:23:58.189968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:58.189923 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" event={"ID":"e164cd97-6fd0-49c6-b20d-9f30bffaca59","Type":"ContainerStarted","Data":"63353de7cdc39d8e23a9ebcdac59b6d0f30e7e24124e5b429ede3eeba745a17b"} Oct 31 19:23:58.405574 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:58.405545 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-pdmbp"] Oct 31 19:23:58.408228 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:23:58.408192 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0853112_0260_4a67_93d3_b885b7944089.slice/crio-446bee6dd9d608c105b6621ef04058b8d87eedaac3f14b526faa0643d840dae6 WatchSource:0}: Error finding container 446bee6dd9d608c105b6621ef04058b8d87eedaac3f14b526faa0643d840dae6: Status 404 returned error can't find the container with id 446bee6dd9d608c105b6621ef04058b8d87eedaac3f14b526faa0643d840dae6 Oct 31 19:23:59.196705 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:23:59.196664 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-pdmbp" event={"ID":"c0853112-0260-4a67-93d3-b885b7944089","Type":"ContainerStarted","Data":"446bee6dd9d608c105b6621ef04058b8d87eedaac3f14b526faa0643d840dae6"} Oct 31 19:24:01.209873 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:01.209789 4910 generic.go:356] "Generic (PLEG): container finished" podID="e164cd97-6fd0-49c6-b20d-9f30bffaca59" containerID="3d61d07bcc6ad90f133645bbd06d2b77115d84e5e2bd52852a700842aac3e911" exitCode=0 Oct 31 19:24:01.209873 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:01.209848 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" event={"ID":"e164cd97-6fd0-49c6-b20d-9f30bffaca59","Type":"ContainerDied","Data":"3d61d07bcc6ad90f133645bbd06d2b77115d84e5e2bd52852a700842aac3e911"} Oct 31 19:24:02.216479 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:02.216440 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" event={"ID":"e164cd97-6fd0-49c6-b20d-9f30bffaca59","Type":"ContainerStarted","Data":"69b47d6c81f8644aa01ecce7e0fc84bc3ef3f213fe721df72ea07cee4c161e05"} Oct 31 19:24:02.216900 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:02.216511 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" Oct 31 19:24:02.217945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:02.217898 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-pdmbp" event={"ID":"c0853112-0260-4a67-93d3-b885b7944089","Type":"ContainerStarted","Data":"a2399e3270cc91e14387d7211968beaebbbe265665b4b5b746e1fe4f7253f876"} Oct 31 19:24:02.361614 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:02.361557 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" podStartSLOduration=2.425029114 podStartE2EDuration="6.361539228s" totalImagesPullingTime="3.436610522s" totalInitContainerRuntime="499.899592ms" isStatefulPod=false podCreationTimestamp="2026-10-31 19:23:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:02.331859374 +0000 UTC m=+276.694679747" watchObservedRunningTime="2026-10-31 19:24:02.361539228 +0000 UTC m=+276.724359600" Oct 31 19:24:02.362384 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:02.362353 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-pdmbp" podStartSLOduration=2.40547465 podStartE2EDuration="5.362341098s" totalImagesPullingTime="2.956866448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:02.360584747 +0000 UTC m=+276.723405121" watchObservedRunningTime="2026-10-31 19:24:02.362341098 +0000 UTC m=+276.725161470" Oct 31 19:24:02.916575 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:02.916540 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xk49"] Oct 31 19:24:02.920028 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:02.920012 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xk49" Oct 31 19:24:02.929404 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:02.929378 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 19:24:02.929404 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:02.929379 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 19:24:02.929404 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:02.929404 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-f9q55\"" Oct 31 19:24:02.929583 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:02.929548 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67l6p\" (UniqueName: \"kubernetes.io/projected/2bfd6e75-7b11-4c98-aab3-8f3599cdca39-kube-api-access-67l6p\") pod \"dns-operator-controller-manager-8557cbdb84-8xk49\" (UID: \"2bfd6e75-7b11-4c98-aab3-8f3599cdca39\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xk49" Oct 31 19:24:02.932820 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:02.932801 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 31 19:24:02.937444 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:02.937423 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xk49"] Oct 31 19:24:03.030561 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:03.030532 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67l6p\" (UniqueName: \"kubernetes.io/projected/2bfd6e75-7b11-4c98-aab3-8f3599cdca39-kube-api-access-67l6p\") pod \"dns-operator-controller-manager-8557cbdb84-8xk49\" (UID: \"2bfd6e75-7b11-4c98-aab3-8f3599cdca39\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xk49" Oct 31 19:24:03.046811 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:03.046783 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67l6p\" (UniqueName: \"kubernetes.io/projected/2bfd6e75-7b11-4c98-aab3-8f3599cdca39-kube-api-access-67l6p\") pod \"dns-operator-controller-manager-8557cbdb84-8xk49\" (UID: \"2bfd6e75-7b11-4c98-aab3-8f3599cdca39\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xk49" Oct 31 19:24:03.232440 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:03.232365 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xk49" Oct 31 19:24:03.516556 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:03.516531 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xk49"] Oct 31 19:24:03.518509 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:24:03.518483 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bfd6e75_7b11_4c98_aab3_8f3599cdca39.slice/crio-6b3d329037f3c7bdfce077650d0d7b35e5a5288bb114293ef78f2948dc5a4586 WatchSource:0}: Error finding container 6b3d329037f3c7bdfce077650d0d7b35e5a5288bb114293ef78f2948dc5a4586: Status 404 returned error can't find the container with id 6b3d329037f3c7bdfce077650d0d7b35e5a5288bb114293ef78f2948dc5a4586 Oct 31 19:24:04.228161 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:04.228122 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xk49" event={"ID":"2bfd6e75-7b11-4c98-aab3-8f3599cdca39","Type":"ContainerStarted","Data":"6b3d329037f3c7bdfce077650d0d7b35e5a5288bb114293ef78f2948dc5a4586"} Oct 31 19:24:07.245593 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:07.245553 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xk49" event={"ID":"2bfd6e75-7b11-4c98-aab3-8f3599cdca39","Type":"ContainerStarted","Data":"97b03f934cebdb1bfb9f020ed00db2ff43881892649a1f6ffbdae957deaf6c4a"} Oct 31 19:24:07.246004 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:07.245652 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xk49" Oct 31 19:24:07.304126 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:07.304071 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xk49" podStartSLOduration=2.319296353 podStartE2EDuration="5.304057323s" totalImagesPullingTime="2.98476097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:07.297456059 +0000 UTC m=+281.660276431" watchObservedRunningTime="2026-10-31 19:24:07.304057323 +0000 UTC m=+281.666877694" Oct 31 19:24:08.458719 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:08.458687 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-pp8wj"] Oct 31 19:24:08.461983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:08.461966 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-pp8wj" Oct 31 19:24:08.478068 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:08.478044 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8qdc\" (UniqueName: \"kubernetes.io/projected/0c073c51-9ff2-49db-a9f1-3e833a358623-kube-api-access-f8qdc\") pod \"authorino-operator-7b7bb4bc77-pp8wj\" (UID: \"0c073c51-9ff2-49db-a9f1-3e833a358623\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-pp8wj" Oct 31 19:24:08.479700 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:08.479673 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-n68sj\"" Oct 31 19:24:08.490955 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:08.490931 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-pp8wj"] Oct 31 19:24:08.579429 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:08.579395 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8qdc\" (UniqueName: \"kubernetes.io/projected/0c073c51-9ff2-49db-a9f1-3e833a358623-kube-api-access-f8qdc\") pod \"authorino-operator-7b7bb4bc77-pp8wj\" (UID: \"0c073c51-9ff2-49db-a9f1-3e833a358623\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-pp8wj" Oct 31 19:24:08.598185 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:08.598158 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8qdc\" (UniqueName: \"kubernetes.io/projected/0c073c51-9ff2-49db-a9f1-3e833a358623-kube-api-access-f8qdc\") pod \"authorino-operator-7b7bb4bc77-pp8wj\" (UID: \"0c073c51-9ff2-49db-a9f1-3e833a358623\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-pp8wj" Oct 31 19:24:08.774189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:08.774108 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-pp8wj" Oct 31 19:24:09.077771 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:09.077750 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-pp8wj"] Oct 31 19:24:09.079440 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:24:09.079405 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c073c51_9ff2_49db_a9f1_3e833a358623.slice/crio-c5ece3bb53379cdfdb7dd71a9c9f636c1d44459fdf80ceca6ad775f114360cb6 WatchSource:0}: Error finding container c5ece3bb53379cdfdb7dd71a9c9f636c1d44459fdf80ceca6ad775f114360cb6: Status 404 returned error can't find the container with id c5ece3bb53379cdfdb7dd71a9c9f636c1d44459fdf80ceca6ad775f114360cb6 Oct 31 19:24:09.255258 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:09.255224 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-pp8wj" event={"ID":"0c073c51-9ff2-49db-a9f1-3e833a358623","Type":"ContainerStarted","Data":"c5ece3bb53379cdfdb7dd71a9c9f636c1d44459fdf80ceca6ad775f114360cb6"} Oct 31 19:24:11.266086 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:11.266048 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-pp8wj" event={"ID":"0c073c51-9ff2-49db-a9f1-3e833a358623","Type":"ContainerStarted","Data":"a2cdc0a66a23ed0106691c7090b69372293422414f87e743f875ed4428520530"} Oct 31 19:24:11.266459 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:11.266134 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-pp8wj" Oct 31 19:24:11.302872 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:11.302819 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-pp8wj" podStartSLOduration=1.288152231 podStartE2EDuration="3.302805473s" totalImagesPullingTime="2.014653242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:11.300064166 +0000 UTC m=+285.662884542" watchObservedRunningTime="2026-10-31 19:24:11.302805473 +0000 UTC m=+285.665625845" Oct 31 19:24:12.712748 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:24:12.712705 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:24:12.780747 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:12.780708 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-q5rgw"] Oct 31 19:24:12.785494 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:12.785476 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q5rgw" Oct 31 19:24:12.797121 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:12.797103 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-nvgtq\"" Oct 31 19:24:12.802320 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:12.802286 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-q5rgw"] Oct 31 19:24:12.816770 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:12.816749 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpggq\" (UniqueName: \"kubernetes.io/projected/d7afd221-6685-485c-933a-e3ac3b5e701b-kube-api-access-rpggq\") pod \"limitador-operator-controller-manager-5778bb4449-q5rgw\" (UID: \"d7afd221-6685-485c-933a-e3ac3b5e701b\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q5rgw" Oct 31 19:24:12.917678 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:12.917629 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpggq\" (UniqueName: \"kubernetes.io/projected/d7afd221-6685-485c-933a-e3ac3b5e701b-kube-api-access-rpggq\") pod \"limitador-operator-controller-manager-5778bb4449-q5rgw\" (UID: \"d7afd221-6685-485c-933a-e3ac3b5e701b\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q5rgw" Oct 31 19:24:12.935038 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:12.935006 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpggq\" (UniqueName: \"kubernetes.io/projected/d7afd221-6685-485c-933a-e3ac3b5e701b-kube-api-access-rpggq\") pod \"limitador-operator-controller-manager-5778bb4449-q5rgw\" (UID: \"d7afd221-6685-485c-933a-e3ac3b5e701b\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q5rgw" Oct 31 19:24:13.098690 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:13.098655 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q5rgw" Oct 31 19:24:13.225266 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:13.225236 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-ptzqm" Oct 31 19:24:13.379500 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:13.379468 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-q5rgw"] Oct 31 19:24:13.380334 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:24:13.380306 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7afd221_6685_485c_933a_e3ac3b5e701b.slice/crio-e1e8a360725e3360930c9f042ad6171528bd517ee8587a30d89efcac1b5afeca WatchSource:0}: Error finding container e1e8a360725e3360930c9f042ad6171528bd517ee8587a30d89efcac1b5afeca: Status 404 returned error can't find the container with id e1e8a360725e3360930c9f042ad6171528bd517ee8587a30d89efcac1b5afeca Oct 31 19:24:14.283624 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:14.283589 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q5rgw" event={"ID":"d7afd221-6685-485c-933a-e3ac3b5e701b","Type":"ContainerStarted","Data":"e1e8a360725e3360930c9f042ad6171528bd517ee8587a30d89efcac1b5afeca"} Oct 31 19:24:16.293762 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:16.293722 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q5rgw" event={"ID":"d7afd221-6685-485c-933a-e3ac3b5e701b","Type":"ContainerStarted","Data":"7258bde994bb86dda41da3e42f6bd198a474a92e2e85f1443b1bcb00ae9fc7d5"} Oct 31 19:24:16.294259 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:16.293807 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q5rgw" Oct 31 19:24:16.327209 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:16.327157 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q5rgw" podStartSLOduration=2.449290787 podStartE2EDuration="4.327141096s" totalImagesPullingTime="1.877850309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:16.326052802 +0000 UTC m=+290.688873174" watchObservedRunningTime="2026-10-31 19:24:16.327141096 +0000 UTC m=+290.689961468" Oct 31 19:24:18.253029 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:18.252997 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-8xk49" Oct 31 19:24:19.358202 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:19.358165 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m6g9g"] Oct 31 19:24:19.378599 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:19.378566 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m6g9g" Oct 31 19:24:19.402258 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:19.402229 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m6g9g"] Oct 31 19:24:19.436398 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:19.436373 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vj5f2\"" Oct 31 19:24:19.475589 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:19.475558 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26kdr\" (UniqueName: \"kubernetes.io/projected/54458929-81d7-4343-af60-fa19929a513e-kube-api-access-26kdr\") pod \"kuadrant-operator-controller-manager-678d87766f-m6g9g\" (UID: \"54458929-81d7-4343-af60-fa19929a513e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m6g9g" Oct 31 19:24:19.475722 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:19.475622 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/54458929-81d7-4343-af60-fa19929a513e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-m6g9g\" (UID: \"54458929-81d7-4343-af60-fa19929a513e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m6g9g" Oct 31 19:24:19.576804 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:19.576771 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26kdr\" (UniqueName: \"kubernetes.io/projected/54458929-81d7-4343-af60-fa19929a513e-kube-api-access-26kdr\") pod \"kuadrant-operator-controller-manager-678d87766f-m6g9g\" (UID: \"54458929-81d7-4343-af60-fa19929a513e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m6g9g" Oct 31 19:24:19.576951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:19.576831 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/54458929-81d7-4343-af60-fa19929a513e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-m6g9g\" (UID: \"54458929-81d7-4343-af60-fa19929a513e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m6g9g" Oct 31 19:24:19.577199 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:19.577179 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/54458929-81d7-4343-af60-fa19929a513e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-m6g9g\" (UID: \"54458929-81d7-4343-af60-fa19929a513e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m6g9g" Oct 31 19:24:19.594511 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:19.594486 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26kdr\" (UniqueName: \"kubernetes.io/projected/54458929-81d7-4343-af60-fa19929a513e-kube-api-access-26kdr\") pod \"kuadrant-operator-controller-manager-678d87766f-m6g9g\" (UID: \"54458929-81d7-4343-af60-fa19929a513e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m6g9g" Oct 31 19:24:19.691063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:19.690980 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m6g9g" Oct 31 19:24:20.000869 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:20.000839 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m6g9g"] Oct 31 19:24:20.002358 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:24:20.002323 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54458929_81d7_4343_af60_fa19929a513e.slice/crio-3d4023d5faf3adb1b53aed6a4c6fb9088362180d2af07f5f8f100d93ffea9c8c WatchSource:0}: Error finding container 3d4023d5faf3adb1b53aed6a4c6fb9088362180d2af07f5f8f100d93ffea9c8c: Status 404 returned error can't find the container with id 3d4023d5faf3adb1b53aed6a4c6fb9088362180d2af07f5f8f100d93ffea9c8c Oct 31 19:24:20.314162 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:20.314126 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m6g9g" event={"ID":"54458929-81d7-4343-af60-fa19929a513e","Type":"ContainerStarted","Data":"3d4023d5faf3adb1b53aed6a4c6fb9088362180d2af07f5f8f100d93ffea9c8c"} Oct 31 19:24:22.272870 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:22.272834 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-pp8wj" Oct 31 19:24:24.825330 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:24.823103 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8"] Oct 31 19:24:24.833158 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:24.832728 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:24.845692 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:24.845663 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-784xh\"" Oct 31 19:24:24.848867 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:24.848838 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8"] Oct 31 19:24:24.927611 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:24.927574 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-752rf\" (UniqueName: \"kubernetes.io/projected/23f65946-0acc-4386-89a8-3d350599d6ee-kube-api-access-752rf\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:24.927791 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:24.927650 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f65946-0acc-4386-89a8-3d350599d6ee-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:24.927791 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:24.927735 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f65946-0acc-4386-89a8-3d350599d6ee-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:24.927791 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:24.927776 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f65946-0acc-4386-89a8-3d350599d6ee-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:24.928020 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:24.927888 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f65946-0acc-4386-89a8-3d350599d6ee-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:24.928020 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:24.927950 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f65946-0acc-4386-89a8-3d350599d6ee-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:24.928020 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:24.928006 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f65946-0acc-4386-89a8-3d350599d6ee-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:24.928140 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:24.928045 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f65946-0acc-4386-89a8-3d350599d6ee-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:24.928140 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:24.928088 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f65946-0acc-4386-89a8-3d350599d6ee-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:25.028746 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:25.028704 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f65946-0acc-4386-89a8-3d350599d6ee-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:25.028927 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:25.028757 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f65946-0acc-4386-89a8-3d350599d6ee-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:25.028927 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:25.028807 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f65946-0acc-4386-89a8-3d350599d6ee-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:25.028927 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:25.028832 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f65946-0acc-4386-89a8-3d350599d6ee-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:25.028927 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:25.028882 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f65946-0acc-4386-89a8-3d350599d6ee-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:25.029087 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:25.028958 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f65946-0acc-4386-89a8-3d350599d6ee-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:25.029087 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:25.029006 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f65946-0acc-4386-89a8-3d350599d6ee-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:25.029087 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:25.029049 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-752rf\" (UniqueName: \"kubernetes.io/projected/23f65946-0acc-4386-89a8-3d350599d6ee-kube-api-access-752rf\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:25.029198 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:25.029104 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f65946-0acc-4386-89a8-3d350599d6ee-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:25.029198 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:25.029134 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23f65946-0acc-4386-89a8-3d350599d6ee-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:25.029400 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:25.029382 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23f65946-0acc-4386-89a8-3d350599d6ee-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:25.031141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:25.029882 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f65946-0acc-4386-89a8-3d350599d6ee-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:25.031141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:25.030013 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23f65946-0acc-4386-89a8-3d350599d6ee-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:25.031141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:25.030095 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23f65946-0acc-4386-89a8-3d350599d6ee-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:25.031659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:25.031636 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23f65946-0acc-4386-89a8-3d350599d6ee-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:25.032186 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:25.032166 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23f65946-0acc-4386-89a8-3d350599d6ee-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:25.044388 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:25.044360 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23f65946-0acc-4386-89a8-3d350599d6ee-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:25.054684 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:25.054656 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-752rf\" (UniqueName: \"kubernetes.io/projected/23f65946-0acc-4386-89a8-3d350599d6ee-kube-api-access-752rf\") pod \"data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8\" (UID: \"23f65946-0acc-4386-89a8-3d350599d6ee\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:25.153251 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:25.153164 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:25.447889 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:25.447843 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8"] Oct 31 19:24:25.450344 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:24:25.450315 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f65946_0acc_4386_89a8_3d350599d6ee.slice/crio-5a4231775e9dbc17c3a6d4f71e3b2bdb1e7368dadac7f3c9686d3bcfe7cd41b9 WatchSource:0}: Error finding container 5a4231775e9dbc17c3a6d4f71e3b2bdb1e7368dadac7f3c9686d3bcfe7cd41b9: Status 404 returned error can't find the container with id 5a4231775e9dbc17c3a6d4f71e3b2bdb1e7368dadac7f3c9686d3bcfe7cd41b9 Oct 31 19:24:25.452856 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:25.452817 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:24:25.452947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:25.452892 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:24:25.452947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:25.452942 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:24:25.792072 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:25.791993 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 19:24:25.792215 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:25.792128 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 19:24:26.350869 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:26.350825 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" event={"ID":"23f65946-0acc-4386-89a8-3d350599d6ee","Type":"ContainerStarted","Data":"4dd447419d67f4d7e8919cbb97e32071a98081e9b65feb811fd1a24c8b294cbc"} Oct 31 19:24:26.351363 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:26.350886 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" event={"ID":"23f65946-0acc-4386-89a8-3d350599d6ee","Type":"ContainerStarted","Data":"5a4231775e9dbc17c3a6d4f71e3b2bdb1e7368dadac7f3c9686d3bcfe7cd41b9"} Oct 31 19:24:26.352647 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:26.352626 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m6g9g" event={"ID":"54458929-81d7-4343-af60-fa19929a513e","Type":"ContainerStarted","Data":"ea3bcc90e7e40213921afb9dbc358d51cfd6967108122081a420f19429e47f61"} Oct 31 19:24:26.352773 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:26.352759 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m6g9g" Oct 31 19:24:26.397625 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:26.397559 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" podStartSLOduration=2.397543597 podStartE2EDuration="2.397543597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:26.395236699 +0000 UTC m=+300.758057097" watchObservedRunningTime="2026-10-31 19:24:26.397543597 +0000 UTC m=+300.760363968" Oct 31 19:24:26.431494 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:26.431434 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m6g9g" podStartSLOduration=1.91950472 podStartE2EDuration="7.431413413s" totalImagesPullingTime="5.511908693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:26.427570407 +0000 UTC m=+300.790390779" watchObservedRunningTime="2026-10-31 19:24:26.431413413 +0000 UTC m=+300.794233787" Oct 31 19:24:27.153959 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:27.153899 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:27.158951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:27.158903 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:27.301428 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:27.301402 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-q5rgw" Oct 31 19:24:27.359696 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:27.359609 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:27.361026 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:27.361001 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8" Oct 31 19:24:27.387214 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:27.387182 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-cx5d8"] Oct 31 19:24:27.392143 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:27.392117 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cx5d8" Oct 31 19:24:27.402295 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:27.402269 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 31 19:24:27.411759 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:27.411701 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-cx5d8"] Oct 31 19:24:27.435664 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:27.435457 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 31 19:24:27.435664 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:27.435577 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mwdrn\"" Oct 31 19:24:27.452735 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:27.452705 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b103a6dc-3af0-4ae2-b3fb-5c7b82042013-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-cx5d8\" (UID: \"b103a6dc-3af0-4ae2-b3fb-5c7b82042013\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cx5d8" Oct 31 19:24:27.452906 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:27.452741 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vldxh\" (UniqueName: \"kubernetes.io/projected/b103a6dc-3af0-4ae2-b3fb-5c7b82042013-kube-api-access-vldxh\") pod \"kuadrant-console-plugin-66c4fb575d-cx5d8\" (UID: \"b103a6dc-3af0-4ae2-b3fb-5c7b82042013\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cx5d8" Oct 31 19:24:27.452906 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:27.452834 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b103a6dc-3af0-4ae2-b3fb-5c7b82042013-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-cx5d8\" (UID: \"b103a6dc-3af0-4ae2-b3fb-5c7b82042013\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cx5d8" Oct 31 19:24:27.554019 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:27.553984 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b103a6dc-3af0-4ae2-b3fb-5c7b82042013-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-cx5d8\" (UID: \"b103a6dc-3af0-4ae2-b3fb-5c7b82042013\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cx5d8" Oct 31 19:24:27.554019 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:27.554021 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vldxh\" (UniqueName: \"kubernetes.io/projected/b103a6dc-3af0-4ae2-b3fb-5c7b82042013-kube-api-access-vldxh\") pod \"kuadrant-console-plugin-66c4fb575d-cx5d8\" (UID: \"b103a6dc-3af0-4ae2-b3fb-5c7b82042013\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cx5d8" Oct 31 19:24:27.554229 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:27.554054 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b103a6dc-3af0-4ae2-b3fb-5c7b82042013-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-cx5d8\" (UID: \"b103a6dc-3af0-4ae2-b3fb-5c7b82042013\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cx5d8" Oct 31 19:24:27.554768 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:27.554746 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b103a6dc-3af0-4ae2-b3fb-5c7b82042013-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-cx5d8\" (UID: \"b103a6dc-3af0-4ae2-b3fb-5c7b82042013\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cx5d8" Oct 31 19:24:27.556490 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:27.556449 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b103a6dc-3af0-4ae2-b3fb-5c7b82042013-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-cx5d8\" (UID: \"b103a6dc-3af0-4ae2-b3fb-5c7b82042013\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cx5d8" Oct 31 19:24:27.590431 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:27.590406 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vldxh\" (UniqueName: \"kubernetes.io/projected/b103a6dc-3af0-4ae2-b3fb-5c7b82042013-kube-api-access-vldxh\") pod \"kuadrant-console-plugin-66c4fb575d-cx5d8\" (UID: \"b103a6dc-3af0-4ae2-b3fb-5c7b82042013\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cx5d8" Oct 31 19:24:27.707390 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:27.707307 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cx5d8" Oct 31 19:24:27.994130 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:27.994093 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-cx5d8"] Oct 31 19:24:27.995271 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:24:27.995243 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb103a6dc_3af0_4ae2_b3fb_5c7b82042013.slice/crio-524912a5668f93c8d3b9a4f16dc534d7e4a15f5413ef1bde5934e10dbb7f297f WatchSource:0}: Error finding container 524912a5668f93c8d3b9a4f16dc534d7e4a15f5413ef1bde5934e10dbb7f297f: Status 404 returned error can't find the container with id 524912a5668f93c8d3b9a4f16dc534d7e4a15f5413ef1bde5934e10dbb7f297f Oct 31 19:24:27.996704 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:27.996687 4910 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 19:24:28.158505 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:28.158468 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9z4zz"] Oct 31 19:24:28.163516 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:28.163495 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-9z4zz" Oct 31 19:24:28.176131 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:28.176108 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9z4zz"] Oct 31 19:24:28.177275 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:28.177254 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-42cfq\"" Oct 31 19:24:28.262440 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:28.262355 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sprqj\" (UniqueName: \"kubernetes.io/projected/cc695b0e-de13-4ac9-b7ca-8c716a5be216-kube-api-access-sprqj\") pod \"authorino-56c6c9ddd5-9z4zz\" (UID: \"cc695b0e-de13-4ac9-b7ca-8c716a5be216\") " pod="kuadrant-system/authorino-56c6c9ddd5-9z4zz" Oct 31 19:24:28.362862 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:28.362822 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sprqj\" (UniqueName: \"kubernetes.io/projected/cc695b0e-de13-4ac9-b7ca-8c716a5be216-kube-api-access-sprqj\") pod \"authorino-56c6c9ddd5-9z4zz\" (UID: \"cc695b0e-de13-4ac9-b7ca-8c716a5be216\") " pod="kuadrant-system/authorino-56c6c9ddd5-9z4zz" Oct 31 19:24:28.366253 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:28.366218 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cx5d8" event={"ID":"b103a6dc-3af0-4ae2-b3fb-5c7b82042013","Type":"ContainerStarted","Data":"524912a5668f93c8d3b9a4f16dc534d7e4a15f5413ef1bde5934e10dbb7f297f"} Oct 31 19:24:28.383506 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:28.383479 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sprqj\" (UniqueName: \"kubernetes.io/projected/cc695b0e-de13-4ac9-b7ca-8c716a5be216-kube-api-access-sprqj\") pod \"authorino-56c6c9ddd5-9z4zz\" (UID: \"cc695b0e-de13-4ac9-b7ca-8c716a5be216\") " pod="kuadrant-system/authorino-56c6c9ddd5-9z4zz" Oct 31 19:24:28.475568 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:28.475533 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-9z4zz" Oct 31 19:24:28.747752 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:28.747729 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9z4zz"] Oct 31 19:24:28.749600 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:24:28.749563 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc695b0e_de13_4ac9_b7ca_8c716a5be216.slice/crio-dd89ed2d5f4f64c67547f191cfb6ab9bd970b6324be243ee4ad1893888b69a1a WatchSource:0}: Error finding container dd89ed2d5f4f64c67547f191cfb6ab9bd970b6324be243ee4ad1893888b69a1a: Status 404 returned error can't find the container with id dd89ed2d5f4f64c67547f191cfb6ab9bd970b6324be243ee4ad1893888b69a1a Oct 31 19:24:29.377273 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:29.377229 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-9z4zz" event={"ID":"cc695b0e-de13-4ac9-b7ca-8c716a5be216","Type":"ContainerStarted","Data":"dd89ed2d5f4f64c67547f191cfb6ab9bd970b6324be243ee4ad1893888b69a1a"} Oct 31 19:24:31.392360 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:31.392315 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-9z4zz" event={"ID":"cc695b0e-de13-4ac9-b7ca-8c716a5be216","Type":"ContainerStarted","Data":"6526a007632afb3b273d91f8341c058887adff7ebdd2ab9338c0e452295baecd"} Oct 31 19:24:31.420954 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:31.420887 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-9z4zz" podStartSLOduration=0.932212169 podStartE2EDuration="3.420872765s" totalImagesPullingTime="2.488660596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:31.419553981 +0000 UTC m=+305.782374354" watchObservedRunningTime="2026-10-31 19:24:31.420872765 +0000 UTC m=+305.783693131" Oct 31 19:24:34.410495 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:34.410457 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cx5d8" event={"ID":"b103a6dc-3af0-4ae2-b3fb-5c7b82042013","Type":"ContainerStarted","Data":"1a061071f0d9c4c5e8a50f2118f13257bdf75cfa07d0b5cc10faf10307b0e584"} Oct 31 19:24:34.430489 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:34.430442 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-cx5d8" podStartSLOduration=1.931230201 podStartE2EDuration="7.430428461s" totalImagesPullingTime="5.49919826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:34.428460219 +0000 UTC m=+308.791280590" watchObservedRunningTime="2026-10-31 19:24:34.430428461 +0000 UTC m=+308.793248857" Oct 31 19:24:37.361905 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:24:37.361824 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m6g9g" Oct 31 19:24:42.712882 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:24:42.712842 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:25:12.712516 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:25:12.712475 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:25:20.850892 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:20.850862 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-bx5w4"] Oct 31 19:25:20.854418 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:20.854398 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bx5w4" Oct 31 19:25:20.862058 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:20.862037 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 31 19:25:20.863685 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:20.863660 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 19:25:20.863941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:20.863926 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 19:25:20.866340 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:20.866317 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-bx5w4"] Oct 31 19:25:20.866771 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:20.866754 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-t2hhj\"" Oct 31 19:25:20.879690 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:20.879666 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-lz987"] Oct 31 19:25:20.883492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:20.883475 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-lz987" Oct 31 19:25:20.896150 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:20.896126 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-lz987"] Oct 31 19:25:20.920684 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:20.920646 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77s58\" (UniqueName: \"kubernetes.io/projected/86198f51-80c5-42f0-a893-112b18a971ea-kube-api-access-77s58\") pod \"llmisvc-controller-manager-869d5dcc5d-bx5w4\" (UID: \"86198f51-80c5-42f0-a893-112b18a971ea\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bx5w4" Oct 31 19:25:20.920851 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:20.920775 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86198f51-80c5-42f0-a893-112b18a971ea-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-bx5w4\" (UID: \"86198f51-80c5-42f0-a893-112b18a971ea\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bx5w4" Oct 31 19:25:20.961160 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:20.961132 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-dbsr2\"" Oct 31 19:25:20.961160 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:20.961132 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 31 19:25:21.021457 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:21.021427 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77s58\" (UniqueName: \"kubernetes.io/projected/86198f51-80c5-42f0-a893-112b18a971ea-kube-api-access-77s58\") pod \"llmisvc-controller-manager-869d5dcc5d-bx5w4\" (UID: \"86198f51-80c5-42f0-a893-112b18a971ea\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bx5w4" Oct 31 19:25:21.021634 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:21.021488 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb8qg\" (UniqueName: \"kubernetes.io/projected/35fff0bc-d871-4a3b-a6ab-0b6616c80dbd-kube-api-access-fb8qg\") pod \"model-serving-api-75bd89fdfc-lz987\" (UID: \"35fff0bc-d871-4a3b-a6ab-0b6616c80dbd\") " pod="opendatahub/model-serving-api-75bd89fdfc-lz987" Oct 31 19:25:21.021634 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:21.021510 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86198f51-80c5-42f0-a893-112b18a971ea-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-bx5w4\" (UID: \"86198f51-80c5-42f0-a893-112b18a971ea\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bx5w4" Oct 31 19:25:21.021722 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:21.021643 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35fff0bc-d871-4a3b-a6ab-0b6616c80dbd-tls-certs\") pod \"model-serving-api-75bd89fdfc-lz987\" (UID: \"35fff0bc-d871-4a3b-a6ab-0b6616c80dbd\") " pod="opendatahub/model-serving-api-75bd89fdfc-lz987" Oct 31 19:25:21.023882 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:21.023856 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86198f51-80c5-42f0-a893-112b18a971ea-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-bx5w4\" (UID: \"86198f51-80c5-42f0-a893-112b18a971ea\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bx5w4" Oct 31 19:25:21.034078 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:21.034054 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77s58\" (UniqueName: \"kubernetes.io/projected/86198f51-80c5-42f0-a893-112b18a971ea-kube-api-access-77s58\") pod \"llmisvc-controller-manager-869d5dcc5d-bx5w4\" (UID: \"86198f51-80c5-42f0-a893-112b18a971ea\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bx5w4" Oct 31 19:25:21.122670 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:21.122579 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35fff0bc-d871-4a3b-a6ab-0b6616c80dbd-tls-certs\") pod \"model-serving-api-75bd89fdfc-lz987\" (UID: \"35fff0bc-d871-4a3b-a6ab-0b6616c80dbd\") " pod="opendatahub/model-serving-api-75bd89fdfc-lz987" Oct 31 19:25:21.122827 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:21.122675 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb8qg\" (UniqueName: \"kubernetes.io/projected/35fff0bc-d871-4a3b-a6ab-0b6616c80dbd-kube-api-access-fb8qg\") pod \"model-serving-api-75bd89fdfc-lz987\" (UID: \"35fff0bc-d871-4a3b-a6ab-0b6616c80dbd\") " pod="opendatahub/model-serving-api-75bd89fdfc-lz987" Oct 31 19:25:21.125133 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:21.125112 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35fff0bc-d871-4a3b-a6ab-0b6616c80dbd-tls-certs\") pod \"model-serving-api-75bd89fdfc-lz987\" (UID: \"35fff0bc-d871-4a3b-a6ab-0b6616c80dbd\") " pod="opendatahub/model-serving-api-75bd89fdfc-lz987" Oct 31 19:25:21.139522 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:21.139497 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb8qg\" (UniqueName: \"kubernetes.io/projected/35fff0bc-d871-4a3b-a6ab-0b6616c80dbd-kube-api-access-fb8qg\") pod \"model-serving-api-75bd89fdfc-lz987\" (UID: \"35fff0bc-d871-4a3b-a6ab-0b6616c80dbd\") " pod="opendatahub/model-serving-api-75bd89fdfc-lz987" Oct 31 19:25:21.168415 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:21.168389 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bx5w4" Oct 31 19:25:21.196284 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:21.196255 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-lz987" Oct 31 19:25:21.449644 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:21.449620 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-bx5w4"] Oct 31 19:25:21.450666 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:25:21.450642 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod86198f51_80c5_42f0_a893_112b18a971ea.slice/crio-703f24c4dc7afc129744689b837fbbecd77dc798f388511bed2b9441deb970e8 WatchSource:0}: Error finding container 703f24c4dc7afc129744689b837fbbecd77dc798f388511bed2b9441deb970e8: Status 404 returned error can't find the container with id 703f24c4dc7afc129744689b837fbbecd77dc798f388511bed2b9441deb970e8 Oct 31 19:25:21.479271 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:21.479246 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-lz987"] Oct 31 19:25:21.480630 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:25:21.480603 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35fff0bc_d871_4a3b_a6ab_0b6616c80dbd.slice/crio-568625bb8ed379b27180882d64c09bf13b374e275c9c5fa1a06e5a06562e8c8a WatchSource:0}: Error finding container 568625bb8ed379b27180882d64c09bf13b374e275c9c5fa1a06e5a06562e8c8a: Status 404 returned error can't find the container with id 568625bb8ed379b27180882d64c09bf13b374e275c9c5fa1a06e5a06562e8c8a Oct 31 19:25:21.639612 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:21.639575 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bx5w4" event={"ID":"86198f51-80c5-42f0-a893-112b18a971ea","Type":"ContainerStarted","Data":"703f24c4dc7afc129744689b837fbbecd77dc798f388511bed2b9441deb970e8"} Oct 31 19:25:21.640675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:21.640653 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-lz987" event={"ID":"35fff0bc-d871-4a3b-a6ab-0b6616c80dbd","Type":"ContainerStarted","Data":"568625bb8ed379b27180882d64c09bf13b374e275c9c5fa1a06e5a06562e8c8a"} Oct 31 19:25:24.661985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:24.661947 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-lz987" event={"ID":"35fff0bc-d871-4a3b-a6ab-0b6616c80dbd","Type":"ContainerStarted","Data":"8b764efa9861ecfbaf0987eb22353e76cf1ac2fcfb65b7d6acab927d2ea864a2"} Oct 31 19:25:24.662389 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:24.662018 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-lz987" Oct 31 19:25:24.663491 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:24.663465 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bx5w4" event={"ID":"86198f51-80c5-42f0-a893-112b18a971ea","Type":"ContainerStarted","Data":"6461d5d3792ac07fdfe20b07264b9774a4a8a117c53b00a6b88468135b24b732"} Oct 31 19:25:24.663602 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:24.663551 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bx5w4" Oct 31 19:25:24.681690 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:24.681643 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-lz987" podStartSLOduration=2.508875431 podStartE2EDuration="4.681630448s" totalImagesPullingTime="2.172755017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:24.68132312 +0000 UTC m=+359.044143491" watchObservedRunningTime="2026-10-31 19:25:24.681630448 +0000 UTC m=+359.044450820" Oct 31 19:25:24.702570 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:24.702518 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bx5w4" podStartSLOduration=1.664476729 podStartE2EDuration="4.702504074s" totalImagesPullingTime="3.038027345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:24.700866778 +0000 UTC m=+359.063687151" watchObservedRunningTime="2026-10-31 19:25:24.702504074 +0000 UTC m=+359.065324445" Oct 31 19:25:35.673876 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:35.673845 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-lz987" Oct 31 19:25:42.711136 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:25:42.711098 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:25:50.796218 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:50.796182 4910 generic.go:356] "Generic (PLEG): container finished" podID="f5b43015-708e-4ef2-883e-cc121fe1c8e5" containerID="03bde39d6987c97f3e6d8b5a285dfff0a6cfc48bcfd979afabbbb2d535ae0bd9" exitCode=0 Oct 31 19:25:50.796666 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:50.796254 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" event={"ID":"f5b43015-708e-4ef2-883e-cc121fe1c8e5","Type":"ContainerDied","Data":"03bde39d6987c97f3e6d8b5a285dfff0a6cfc48bcfd979afabbbb2d535ae0bd9"} Oct 31 19:25:50.796666 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:50.796638 4910 scope.go:122] "RemoveContainer" containerID="03bde39d6987c97f3e6d8b5a285dfff0a6cfc48bcfd979afabbbb2d535ae0bd9" Oct 31 19:25:50.797702 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:50.797674 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:25:50.797752 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:50.797740 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:25:50.797785 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:50.797775 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:25:51.802797 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:51.802765 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" event={"ID":"f5b43015-708e-4ef2-883e-cc121fe1c8e5","Type":"ContainerStarted","Data":"196faf2e816ca4beb2f470a871b5384af6659b277d1a51cfb4d413faba86e1e2"} Oct 31 19:25:51.857021 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:51.856984 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:25:51.857287 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:51.857263 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-w7w5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Oct 31 19:25:51.857384 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:51.857325 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" podUID="f5b43015-708e-4ef2-883e-cc121fe1c8e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Oct 31 19:25:52.856990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:52.856954 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-w7w5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Oct 31 19:25:52.857382 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:52.857012 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" podUID="f5b43015-708e-4ef2-883e-cc121fe1c8e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Oct 31 19:25:53.263546 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:53.263451 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9z4zz"] Oct 31 19:25:53.263694 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:53.263670 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-9z4zz" podUID="cc695b0e-de13-4ac9-b7ca-8c716a5be216" containerName="authorino" containerID="cri-o://6526a007632afb3b273d91f8341c058887adff7ebdd2ab9338c0e452295baecd" gracePeriod=30 Oct 31 19:25:53.567612 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:53.567587 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-9z4zz" Oct 31 19:25:53.665346 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:53.665311 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc695b0e-de13-4ac9-b7ca-8c716a5be216-kube-api-access-sprqj\" (UniqueName: \"kubernetes.io/projected/cc695b0e-de13-4ac9-b7ca-8c716a5be216-kube-api-access-sprqj\") pod \"cc695b0e-de13-4ac9-b7ca-8c716a5be216\" (UID: \"cc695b0e-de13-4ac9-b7ca-8c716a5be216\") " Oct 31 19:25:53.667546 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:53.667517 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc695b0e-de13-4ac9-b7ca-8c716a5be216-kube-api-access-sprqj" pod "cc695b0e-de13-4ac9-b7ca-8c716a5be216" (UID: "cc695b0e-de13-4ac9-b7ca-8c716a5be216"). InnerVolumeSpecName "kube-api-access-sprqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:25:53.766366 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:53.766328 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sprqj\" (UniqueName: \"kubernetes.io/projected/cc695b0e-de13-4ac9-b7ca-8c716a5be216-kube-api-access-sprqj\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:25:53.814843 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:53.814809 4910 generic.go:356] "Generic (PLEG): container finished" podID="cc695b0e-de13-4ac9-b7ca-8c716a5be216" containerID="6526a007632afb3b273d91f8341c058887adff7ebdd2ab9338c0e452295baecd" exitCode=0 Oct 31 19:25:53.815018 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:53.814856 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-9z4zz" Oct 31 19:25:53.815018 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:53.814895 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-9z4zz" event={"ID":"cc695b0e-de13-4ac9-b7ca-8c716a5be216","Type":"ContainerDied","Data":"6526a007632afb3b273d91f8341c058887adff7ebdd2ab9338c0e452295baecd"} Oct 31 19:25:53.815018 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:53.814956 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-9z4zz" event={"ID":"cc695b0e-de13-4ac9-b7ca-8c716a5be216","Type":"ContainerDied","Data":"dd89ed2d5f4f64c67547f191cfb6ab9bd970b6324be243ee4ad1893888b69a1a"} Oct 31 19:25:53.815018 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:53.814979 4910 scope.go:122] "RemoveContainer" containerID="6526a007632afb3b273d91f8341c058887adff7ebdd2ab9338c0e452295baecd" Oct 31 19:25:53.850122 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:53.850099 4910 scope.go:122] "RemoveContainer" containerID="6526a007632afb3b273d91f8341c058887adff7ebdd2ab9338c0e452295baecd" Oct 31 19:25:53.850466 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:25:53.850445 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6526a007632afb3b273d91f8341c058887adff7ebdd2ab9338c0e452295baecd\": container with ID starting with 6526a007632afb3b273d91f8341c058887adff7ebdd2ab9338c0e452295baecd not found: ID does not exist" containerID="6526a007632afb3b273d91f8341c058887adff7ebdd2ab9338c0e452295baecd" Oct 31 19:25:53.850540 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:53.850478 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6526a007632afb3b273d91f8341c058887adff7ebdd2ab9338c0e452295baecd"} err="failed to get container status \"6526a007632afb3b273d91f8341c058887adff7ebdd2ab9338c0e452295baecd\": rpc error: code = NotFound desc = could not find container \"6526a007632afb3b273d91f8341c058887adff7ebdd2ab9338c0e452295baecd\": container with ID starting with 6526a007632afb3b273d91f8341c058887adff7ebdd2ab9338c0e452295baecd not found: ID does not exist" Oct 31 19:25:53.855213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:53.855188 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9z4zz"] Oct 31 19:25:53.857168 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:53.857141 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-w7w5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Oct 31 19:25:53.857502 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:53.857191 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" podUID="f5b43015-708e-4ef2-883e-cc121fe1c8e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Oct 31 19:25:53.859226 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:53.859207 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-9z4zz"] Oct 31 19:25:54.857004 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:54.856968 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-w7w5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Oct 31 19:25:54.857208 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:54.857029 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" podUID="f5b43015-708e-4ef2-883e-cc121fe1c8e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Oct 31 19:25:55.670873 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:55.670839 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bx5w4" Oct 31 19:25:55.754644 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:55.754616 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc695b0e-de13-4ac9-b7ca-8c716a5be216" path="/var/lib/kubelet/pods/cc695b0e-de13-4ac9-b7ca-8c716a5be216/volumes" Oct 31 19:25:55.857389 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:55.857357 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-w7w5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Oct 31 19:25:55.857782 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:55.857403 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" podUID="f5b43015-708e-4ef2-883e-cc121fe1c8e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Oct 31 19:25:56.861296 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:56.861267 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:25:56.861670 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:56.861543 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:25:56.862312 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:25:56.862296 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" Oct 31 19:26:12.711670 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:26:12.711581 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:26:34.560504 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:26:34.560468 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-cvgtj"] Oct 31 19:26:34.561087 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:26:34.561020 4910 cpu_manager.go:414] "containerMap: removing container" podUID="cc695b0e-de13-4ac9-b7ca-8c716a5be216" containerName="authorino" Oct 31 19:26:34.561087 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:26:34.561039 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc695b0e-de13-4ac9-b7ca-8c716a5be216" containerName="authorino" Oct 31 19:26:34.561245 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:26:34.561225 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc695b0e-de13-4ac9-b7ca-8c716a5be216" containerName="authorino" Oct 31 19:26:34.566025 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:26:34.565902 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvgtj" Oct 31 19:26:34.574483 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:26:34.574454 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-cvgtj"] Oct 31 19:26:34.608720 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:26:34.608685 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74wmf\" (UniqueName: \"kubernetes.io/projected/2dbfb4a1-52af-48dc-9739-938eff56a72d-kube-api-access-74wmf\") pod \"llmisvc-controller-manager-6466c554d8-cvgtj\" (UID: \"2dbfb4a1-52af-48dc-9739-938eff56a72d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvgtj" Oct 31 19:26:34.608879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:26:34.608729 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2dbfb4a1-52af-48dc-9739-938eff56a72d-cert\") pod \"llmisvc-controller-manager-6466c554d8-cvgtj\" (UID: \"2dbfb4a1-52af-48dc-9739-938eff56a72d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvgtj" Oct 31 19:26:34.709669 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:26:34.709626 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74wmf\" (UniqueName: \"kubernetes.io/projected/2dbfb4a1-52af-48dc-9739-938eff56a72d-kube-api-access-74wmf\") pod \"llmisvc-controller-manager-6466c554d8-cvgtj\" (UID: \"2dbfb4a1-52af-48dc-9739-938eff56a72d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvgtj" Oct 31 19:26:34.709856 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:26:34.709681 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2dbfb4a1-52af-48dc-9739-938eff56a72d-cert\") pod \"llmisvc-controller-manager-6466c554d8-cvgtj\" (UID: \"2dbfb4a1-52af-48dc-9739-938eff56a72d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvgtj" Oct 31 19:26:34.712232 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:26:34.712209 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2dbfb4a1-52af-48dc-9739-938eff56a72d-cert\") pod \"llmisvc-controller-manager-6466c554d8-cvgtj\" (UID: \"2dbfb4a1-52af-48dc-9739-938eff56a72d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvgtj" Oct 31 19:26:34.723443 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:26:34.723409 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74wmf\" (UniqueName: \"kubernetes.io/projected/2dbfb4a1-52af-48dc-9739-938eff56a72d-kube-api-access-74wmf\") pod \"llmisvc-controller-manager-6466c554d8-cvgtj\" (UID: \"2dbfb4a1-52af-48dc-9739-938eff56a72d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvgtj" Oct 31 19:26:34.879657 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:26:34.879556 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvgtj" Oct 31 19:26:35.154282 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:26:35.154253 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-cvgtj"] Oct 31 19:26:35.155604 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:26:35.155576 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dbfb4a1_52af_48dc_9739_938eff56a72d.slice/crio-49934e840ebfacb210b02076e9f2acc2bbc408c327703962ea46b976e6e5ce2a WatchSource:0}: Error finding container 49934e840ebfacb210b02076e9f2acc2bbc408c327703962ea46b976e6e5ce2a: Status 404 returned error can't find the container with id 49934e840ebfacb210b02076e9f2acc2bbc408c327703962ea46b976e6e5ce2a Oct 31 19:26:36.069762 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:26:36.069729 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvgtj" event={"ID":"2dbfb4a1-52af-48dc-9739-938eff56a72d","Type":"ContainerStarted","Data":"0609f496596c07d5977e0c7a4ad4e01d3bec005491b650515df4bac3aa0976b6"} Oct 31 19:26:36.070200 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:26:36.069783 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvgtj" event={"ID":"2dbfb4a1-52af-48dc-9739-938eff56a72d","Type":"ContainerStarted","Data":"49934e840ebfacb210b02076e9f2acc2bbc408c327703962ea46b976e6e5ce2a"} Oct 31 19:26:36.070200 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:26:36.069817 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvgtj" Oct 31 19:26:36.090043 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:26:36.089983 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvgtj" podStartSLOduration=1.725445415 podStartE2EDuration="2.089965639s" totalImagesPullingTime="364.520224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:26:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:26:36.089626442 +0000 UTC m=+430.452446815" watchObservedRunningTime="2026-10-31 19:26:36.089965639 +0000 UTC m=+430.452786012" Oct 31 19:26:42.712261 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:26:42.712223 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:27:07.077720 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:07.077688 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvgtj" Oct 31 19:27:07.146706 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:07.146673 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-bx5w4"] Oct 31 19:27:07.146918 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:07.146871 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bx5w4" podUID="86198f51-80c5-42f0-a893-112b18a971ea" containerName="manager" containerID="cri-o://6461d5d3792ac07fdfe20b07264b9774a4a8a117c53b00a6b88468135b24b732" gracePeriod=30 Oct 31 19:27:07.393516 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:07.393492 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bx5w4" Oct 31 19:27:07.455312 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:07.455281 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86198f51-80c5-42f0-a893-112b18a971ea-kube-api-access-77s58\" (UniqueName: \"kubernetes.io/projected/86198f51-80c5-42f0-a893-112b18a971ea-kube-api-access-77s58\") pod \"86198f51-80c5-42f0-a893-112b18a971ea\" (UID: \"86198f51-80c5-42f0-a893-112b18a971ea\") " Oct 31 19:27:07.455504 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:07.455322 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86198f51-80c5-42f0-a893-112b18a971ea-cert\" (UniqueName: \"kubernetes.io/secret/86198f51-80c5-42f0-a893-112b18a971ea-cert\") pod \"86198f51-80c5-42f0-a893-112b18a971ea\" (UID: \"86198f51-80c5-42f0-a893-112b18a971ea\") " Oct 31 19:27:07.457535 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:07.457507 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86198f51-80c5-42f0-a893-112b18a971ea-kube-api-access-77s58" pod "86198f51-80c5-42f0-a893-112b18a971ea" (UID: "86198f51-80c5-42f0-a893-112b18a971ea"). InnerVolumeSpecName "kube-api-access-77s58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:27:07.457535 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:07.457512 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86198f51-80c5-42f0-a893-112b18a971ea-cert" pod "86198f51-80c5-42f0-a893-112b18a971ea" (UID: "86198f51-80c5-42f0-a893-112b18a971ea"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:27:07.555842 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:07.555810 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77s58\" (UniqueName: \"kubernetes.io/projected/86198f51-80c5-42f0-a893-112b18a971ea-kube-api-access-77s58\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:27:07.555842 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:07.555836 4910 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86198f51-80c5-42f0-a893-112b18a971ea-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:27:08.231006 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:08.230970 4910 generic.go:356] "Generic (PLEG): container finished" podID="86198f51-80c5-42f0-a893-112b18a971ea" containerID="6461d5d3792ac07fdfe20b07264b9774a4a8a117c53b00a6b88468135b24b732" exitCode=0 Oct 31 19:27:08.231399 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:08.231020 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bx5w4" Oct 31 19:27:08.231399 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:08.231055 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bx5w4" event={"ID":"86198f51-80c5-42f0-a893-112b18a971ea","Type":"ContainerDied","Data":"6461d5d3792ac07fdfe20b07264b9774a4a8a117c53b00a6b88468135b24b732"} Oct 31 19:27:08.231399 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:08.231109 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bx5w4" event={"ID":"86198f51-80c5-42f0-a893-112b18a971ea","Type":"ContainerDied","Data":"703f24c4dc7afc129744689b837fbbecd77dc798f388511bed2b9441deb970e8"} Oct 31 19:27:08.231399 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:08.231131 4910 scope.go:122] "RemoveContainer" containerID="6461d5d3792ac07fdfe20b07264b9774a4a8a117c53b00a6b88468135b24b732" Oct 31 19:27:08.239726 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:08.239706 4910 scope.go:122] "RemoveContainer" containerID="6461d5d3792ac07fdfe20b07264b9774a4a8a117c53b00a6b88468135b24b732" Oct 31 19:27:08.239989 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:27:08.239970 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6461d5d3792ac07fdfe20b07264b9774a4a8a117c53b00a6b88468135b24b732\": container with ID starting with 6461d5d3792ac07fdfe20b07264b9774a4a8a117c53b00a6b88468135b24b732 not found: ID does not exist" containerID="6461d5d3792ac07fdfe20b07264b9774a4a8a117c53b00a6b88468135b24b732" Oct 31 19:27:08.240063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:08.239997 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6461d5d3792ac07fdfe20b07264b9774a4a8a117c53b00a6b88468135b24b732"} err="failed to get container status \"6461d5d3792ac07fdfe20b07264b9774a4a8a117c53b00a6b88468135b24b732\": rpc error: code = NotFound desc = could not find container \"6461d5d3792ac07fdfe20b07264b9774a4a8a117c53b00a6b88468135b24b732\": container with ID starting with 6461d5d3792ac07fdfe20b07264b9774a4a8a117c53b00a6b88468135b24b732 not found: ID does not exist" Oct 31 19:27:08.250844 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:08.250817 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-bx5w4"] Oct 31 19:27:08.256695 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:08.256673 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-bx5w4"] Oct 31 19:27:09.757446 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:09.757409 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86198f51-80c5-42f0-a893-112b18a971ea" path="/var/lib/kubelet/pods/86198f51-80c5-42f0-a893-112b18a971ea/volumes" Oct 31 19:27:12.712216 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:27:12.712179 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:27:42.712259 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:27:42.712220 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:27:49.001504 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.001466 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5"] Oct 31 19:27:49.001986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.001841 4910 cpu_manager.go:414] "containerMap: removing container" podUID="86198f51-80c5-42f0-a893-112b18a971ea" containerName="manager" Oct 31 19:27:49.001986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.001853 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86198f51-80c5-42f0-a893-112b18a971ea" containerName="manager" Oct 31 19:27:49.001986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.001935 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="86198f51-80c5-42f0-a893-112b18a971ea" containerName="manager" Oct 31 19:27:49.005191 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.005152 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.018714 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.018692 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5"] Oct 31 19:27:49.038198 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.038163 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.038317 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.038237 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.038381 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.038357 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.038427 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.038406 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.038582 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.038558 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.038678 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.038622 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.038678 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.038651 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5775b1d1-5ece-4ebd-9582-2efad1157d76-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.038785 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.038687 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.038785 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.038725 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2c54\" (UniqueName: \"kubernetes.io/projected/5775b1d1-5ece-4ebd-9582-2efad1157d76-kube-api-access-d2c54\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.139554 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.139516 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.139753 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.139575 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.139753 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.139597 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.139753 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.139638 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.139753 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.139663 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.139753 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.139687 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5775b1d1-5ece-4ebd-9582-2efad1157d76-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.139753 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.139720 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.139753 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.139753 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2c54\" (UniqueName: \"kubernetes.io/projected/5775b1d1-5ece-4ebd-9582-2efad1157d76-kube-api-access-d2c54\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.140196 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.139783 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.140196 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.140060 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.140196 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.140165 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.140334 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.140235 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.140398 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.140376 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.140518 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.140497 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5775b1d1-5ece-4ebd-9582-2efad1157d76-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.142036 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.142008 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.142147 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.142129 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.152668 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.152647 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.152772 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.152719 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2c54\" (UniqueName: \"kubernetes.io/projected/5775b1d1-5ece-4ebd-9582-2efad1157d76-kube-api-access-d2c54\") pod \"maas-default-gateway-openshift-default-75d7c785f-gj6s5\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.319067 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.319032 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:49.602534 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.602503 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5"] Oct 31 19:27:49.603860 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:27:49.603832 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5775b1d1_5ece_4ebd_9582_2efad1157d76.slice/crio-0141c4d2b322b80eb4944c62cbc39c9954ae6b426d3a04accbcef90fc6f20d44 WatchSource:0}: Error finding container 0141c4d2b322b80eb4944c62cbc39c9954ae6b426d3a04accbcef90fc6f20d44: Status 404 returned error can't find the container with id 0141c4d2b322b80eb4944c62cbc39c9954ae6b426d3a04accbcef90fc6f20d44 Oct 31 19:27:49.605990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.605956 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:27:49.606047 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.606025 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:27:49.606085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:49.606054 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:27:50.462800 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:50.462765 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" event={"ID":"5775b1d1-5ece-4ebd-9582-2efad1157d76","Type":"ContainerStarted","Data":"1fcd0786951b5e2577d527f85a080de437a29c6a8aafba18acda0e5c08dbd5ed"} Oct 31 19:27:50.463319 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:50.462828 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" event={"ID":"5775b1d1-5ece-4ebd-9582-2efad1157d76","Type":"ContainerStarted","Data":"0141c4d2b322b80eb4944c62cbc39c9954ae6b426d3a04accbcef90fc6f20d44"} Oct 31 19:27:50.490312 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:50.490263 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" podStartSLOduration=2.490251448 podStartE2EDuration="2.490251448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:27:50.48730609 +0000 UTC m=+504.850126462" watchObservedRunningTime="2026-10-31 19:27:50.490251448 +0000 UTC m=+504.853071819" Oct 31 19:27:51.337936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:51.335808 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:51.337936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:51.335964 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-gj6s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 31 19:27:51.337936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:51.336008 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" podUID="5775b1d1-5ece-4ebd-9582-2efad1157d76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 31 19:27:52.325945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:52.325368 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-gj6s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 31 19:27:52.325945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:52.325431 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" podUID="5775b1d1-5ece-4ebd-9582-2efad1157d76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 31 19:27:53.321402 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:53.321356 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-gj6s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 31 19:27:53.327194 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:53.327159 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" podUID="5775b1d1-5ece-4ebd-9582-2efad1157d76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 31 19:27:54.332941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:54.326302 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-gj6s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 31 19:27:54.332941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:54.326364 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" podUID="5775b1d1-5ece-4ebd-9582-2efad1157d76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 31 19:27:55.330935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:55.325354 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-gj6s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 31 19:27:55.330935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:55.325420 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" podUID="5775b1d1-5ece-4ebd-9582-2efad1157d76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 31 19:27:56.319440 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:56.319404 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-gj6s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 31 19:27:56.319826 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:56.319476 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" podUID="5775b1d1-5ece-4ebd-9582-2efad1157d76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 31 19:27:57.765708 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:57.765664 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:57.766178 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:57.765983 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:27:57.767350 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:27:57.767328 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:28:04.056881 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.056843 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f"] Oct 31 19:28:04.060843 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.060818 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.077270 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.077241 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f"] Oct 31 19:28:04.092125 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.092091 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.092251 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.092146 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.092297 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.092240 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.092297 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.092278 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73a891d-f2de-4099-9a98-bdcbae51888f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.092375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.092303 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhdzc\" (UniqueName: \"kubernetes.io/projected/c73a891d-f2de-4099-9a98-bdcbae51888f-kube-api-access-hhdzc\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.092375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.092357 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.092475 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.092428 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.092531 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.092500 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.092579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.092530 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.193931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.193881 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.193931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.193929 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.194198 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.193994 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.194198 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.194028 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.194198 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.194065 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.194198 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.194098 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73a891d-f2de-4099-9a98-bdcbae51888f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.194198 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.194124 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhdzc\" (UniqueName: \"kubernetes.io/projected/c73a891d-f2de-4099-9a98-bdcbae51888f-kube-api-access-hhdzc\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.194198 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.194177 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.194487 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.194211 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.194487 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.194436 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.194584 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.194491 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.194739 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.194721 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.194799 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.194777 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.194948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.194902 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73a891d-f2de-4099-9a98-bdcbae51888f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.196932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.196888 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.197144 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.197121 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.207733 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.207709 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhdzc\" (UniqueName: \"kubernetes.io/projected/c73a891d-f2de-4099-9a98-bdcbae51888f-kube-api-access-hhdzc\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.207733 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.207717 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-6qm4f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.379094 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.379002 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:04.660200 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.660163 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f"] Oct 31 19:28:04.662008 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:28:04.661982 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73a891d_f2de_4099_9a98_bdcbae51888f.slice/crio-b84503ed694e13a7613cf062cb917c98ae1d91d067f480a91c27071e607c3b3d WatchSource:0}: Error finding container b84503ed694e13a7613cf062cb917c98ae1d91d067f480a91c27071e607c3b3d: Status 404 returned error can't find the container with id b84503ed694e13a7613cf062cb917c98ae1d91d067f480a91c27071e607c3b3d Oct 31 19:28:04.664697 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.664656 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:28:04.664773 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.664744 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:28:04.664812 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:04.664785 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:28:05.551817 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:05.551783 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" event={"ID":"c73a891d-f2de-4099-9a98-bdcbae51888f","Type":"ContainerStarted","Data":"cddbb54274b3fa3ecdd17a9937e1602871e13343544932ac9cbcaca01e01f56d"} Oct 31 19:28:05.552364 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:05.551838 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" event={"ID":"c73a891d-f2de-4099-9a98-bdcbae51888f","Type":"ContainerStarted","Data":"b84503ed694e13a7613cf062cb917c98ae1d91d067f480a91c27071e607c3b3d"} Oct 31 19:28:05.578684 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:05.578636 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" podStartSLOduration=1.578623392 podStartE2EDuration="1.578623392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:28:05.577134271 +0000 UTC m=+519.939954644" watchObservedRunningTime="2026-10-31 19:28:05.578623392 +0000 UTC m=+519.941443764" Oct 31 19:28:06.379290 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:06.379256 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:06.380808 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:06.380780 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6qm4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 31 19:28:06.380950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:06.380834 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" podUID="c73a891d-f2de-4099-9a98-bdcbae51888f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 31 19:28:07.379529 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:07.379492 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6qm4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 31 19:28:07.379967 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:07.379550 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" podUID="c73a891d-f2de-4099-9a98-bdcbae51888f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 31 19:28:08.379566 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:08.379534 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6qm4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 31 19:28:08.380031 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:08.379599 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" podUID="c73a891d-f2de-4099-9a98-bdcbae51888f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 31 19:28:09.383061 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:09.383024 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6qm4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 31 19:28:09.383600 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:09.383078 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" podUID="c73a891d-f2de-4099-9a98-bdcbae51888f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 31 19:28:10.379690 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:10.379657 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6qm4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 31 19:28:10.379878 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:10.379707 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" podUID="c73a891d-f2de-4099-9a98-bdcbae51888f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 31 19:28:11.379756 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:11.379715 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6qm4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 31 19:28:11.380187 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:11.379782 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" podUID="c73a891d-f2de-4099-9a98-bdcbae51888f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 31 19:28:12.712061 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:28:12.712017 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:28:13.380273 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:13.380232 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6qm4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:28:13.380494 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:13.380290 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" podUID="c73a891d-f2de-4099-9a98-bdcbae51888f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:28:14.421137 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:14.421094 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6qm4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:28:14.421622 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:14.421180 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" podUID="c73a891d-f2de-4099-9a98-bdcbae51888f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:28:14.920773 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:14.920741 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:14.921493 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:14.921460 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:14.922308 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:28:14.922286 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:28:42.711611 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:28:42.711570 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:29:02.571857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.571817 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4"] Oct 31 19:29:02.575669 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.575651 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.589301 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.589267 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-8b083a-openshift-default-dockercfg-jr7vh\"" Oct 31 19:29:02.591666 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.591631 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4"] Oct 31 19:29:02.738532 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.738489 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.738710 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.738538 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.738710 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.738578 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.738807 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.738776 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.738843 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.738831 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb65a183-bd28-4a21-8895-82b7a39b4693-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.738880 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.738861 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.738931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.738887 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.738968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.738939 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.739001 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.738963 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrfqm\" (UniqueName: \"kubernetes.io/projected/cb65a183-bd28-4a21-8895-82b7a39b4693-kube-api-access-wrfqm\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.839651 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.839562 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.839651 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.839596 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb65a183-bd28-4a21-8895-82b7a39b4693-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.839651 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.839616 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.839651 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.839640 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.840001 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.839675 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.840001 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.839698 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrfqm\" (UniqueName: \"kubernetes.io/projected/cb65a183-bd28-4a21-8895-82b7a39b4693-kube-api-access-wrfqm\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.840001 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.839764 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.840001 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.839790 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.840001 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.839829 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.840264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.840049 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.840264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.840114 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.840264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.840147 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.840264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.840254 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.841072 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.841047 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb65a183-bd28-4a21-8895-82b7a39b4693-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.842431 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.842402 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.842699 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.842677 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.859762 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.859734 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrfqm\" (UniqueName: \"kubernetes.io/projected/cb65a183-bd28-4a21-8895-82b7a39b4693-kube-api-access-wrfqm\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.866196 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.866170 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.889829 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.889802 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs"] Oct 31 19:29:02.890457 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.890431 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:02.894876 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.894759 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:02.902445 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.902421 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-996c32-openshift-default-dockercfg-lsfd2\"" Oct 31 19:29:02.918103 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:02.918073 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs"] Oct 31 19:29:03.026171 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.026080 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr"] Oct 31 19:29:03.029553 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.029531 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.043419 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.043393 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.043541 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.043428 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.043541 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.043511 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204c09e1-cfce-4add-986d-165e78767839-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.043651 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.043628 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.043719 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.043703 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204c09e1-cfce-4add-986d-165e78767839-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.043794 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.043773 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204c09e1-cfce-4add-986d-165e78767839-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.043832 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.043816 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k7zd\" (UniqueName: \"kubernetes.io/projected/204c09e1-cfce-4add-986d-165e78767839-kube-api-access-4k7zd\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.043877 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.043849 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.043955 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.043938 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.055732 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.055706 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr"] Oct 31 19:29:03.059837 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.059802 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-23843f-openshift-default-dockercfg-27ptb\"" Oct 31 19:29:03.144903 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.144819 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.144903 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.144878 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.145147 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.144961 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.145147 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.145002 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204c09e1-cfce-4add-986d-165e78767839-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.145352 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.145327 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.145425 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.145338 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.145425 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.145402 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16761f60-662e-4eb1-a247-b00378149cc4-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.145425 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.145408 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.145583 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.145433 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.145639 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.145610 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.145695 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.145659 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16761f60-662e-4eb1-a247-b00378149cc4-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.145751 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.145700 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.145751 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.145722 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.145855 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.145799 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204c09e1-cfce-4add-986d-165e78767839-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.145855 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.145842 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204c09e1-cfce-4add-986d-165e78767839-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.145990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.145870 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k7zd\" (UniqueName: \"kubernetes.io/projected/204c09e1-cfce-4add-986d-165e78767839-kube-api-access-4k7zd\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.145990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.145898 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16761f60-662e-4eb1-a247-b00378149cc4-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.145990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.145950 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrvwl\" (UniqueName: \"kubernetes.io/projected/16761f60-662e-4eb1-a247-b00378149cc4-kube-api-access-jrvwl\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.145990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.145980 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.146216 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.146019 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.146216 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.146048 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.146423 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.146391 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.146602 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.146582 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204c09e1-cfce-4add-986d-165e78767839-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.148234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.148205 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204c09e1-cfce-4add-986d-165e78767839-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.148841 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.148818 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.167804 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.167777 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k7zd\" (UniqueName: \"kubernetes.io/projected/204c09e1-cfce-4add-986d-165e78767839-kube-api-access-4k7zd\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.171700 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.171677 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204c09e1-cfce-4add-986d-165e78767839-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.217403 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.217373 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4"] Oct 31 19:29:03.218006 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:29:03.217980 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb65a183_bd28_4a21_8895_82b7a39b4693.slice/crio-232453e97007bfbcb8416b1085daedfa3902cf855f54a7626364fa9f3201f04f WatchSource:0}: Error finding container 232453e97007bfbcb8416b1085daedfa3902cf855f54a7626364fa9f3201f04f: Status 404 returned error can't find the container with id 232453e97007bfbcb8416b1085daedfa3902cf855f54a7626364fa9f3201f04f Oct 31 19:29:03.220937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.220862 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:29:03.221589 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.221032 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:29:03.221589 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.221133 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:29:03.231587 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.231558 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:03.247040 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.247010 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.247108 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.247082 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.247165 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.247124 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16761f60-662e-4eb1-a247-b00378149cc4-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.247165 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.247153 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.247275 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.247194 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16761f60-662e-4eb1-a247-b00378149cc4-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.247275 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.247227 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.247420 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.247273 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16761f60-662e-4eb1-a247-b00378149cc4-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.247420 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.247302 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrvwl\" (UniqueName: \"kubernetes.io/projected/16761f60-662e-4eb1-a247-b00378149cc4-kube-api-access-jrvwl\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.247420 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.247337 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.247570 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.247464 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.247570 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.247475 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.248039 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.248018 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16761f60-662e-4eb1-a247-b00378149cc4-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.248747 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.248725 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.248835 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.248745 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.250236 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.250216 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16761f60-662e-4eb1-a247-b00378149cc4-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.251559 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.251538 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.282863 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.282838 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16761f60-662e-4eb1-a247-b00378149cc4-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.286013 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.285981 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrvwl\" (UniqueName: \"kubernetes.io/projected/16761f60-662e-4eb1-a247-b00378149cc4-kube-api-access-jrvwl\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.344462 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.344191 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:03.533056 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.533026 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs"] Oct 31 19:29:03.533597 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:29:03.533569 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204c09e1_cfce_4add_986d_165e78767839.slice/crio-7eebae0b0db0ac4b03e40f4ffb0f3368042567cb7d80d84d6a31c9db2e8a30c8 WatchSource:0}: Error finding container 7eebae0b0db0ac4b03e40f4ffb0f3368042567cb7d80d84d6a31c9db2e8a30c8: Status 404 returned error can't find the container with id 7eebae0b0db0ac4b03e40f4ffb0f3368042567cb7d80d84d6a31c9db2e8a30c8 Oct 31 19:29:03.536241 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.536202 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:29:03.536333 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.536288 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:29:03.536372 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.536334 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:29:03.700332 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.699984 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr"] Oct 31 19:29:03.711033 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.710994 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:29:03.711123 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.711082 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:29:03.711158 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.711124 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:29:03.850358 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.849440 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" event={"ID":"16761f60-662e-4eb1-a247-b00378149cc4","Type":"ContainerStarted","Data":"3333df1718d6f7235bfac8450c3c102e1e1de86bd13998a7f2a32a7cc5213b4d"} Oct 31 19:29:03.850358 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.849505 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" event={"ID":"16761f60-662e-4eb1-a247-b00378149cc4","Type":"ContainerStarted","Data":"d98f170fdb5c2e31b2005bfa535851c9f33cef65579b3bf69cc76f9a1df047d0"} Oct 31 19:29:03.852572 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.852477 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" event={"ID":"204c09e1-cfce-4add-986d-165e78767839","Type":"ContainerStarted","Data":"de598823f128e0f7ff708756df6ed86a22abc66e59338acd2711304ab9b4a257"} Oct 31 19:29:03.852572 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.852533 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" event={"ID":"204c09e1-cfce-4add-986d-165e78767839","Type":"ContainerStarted","Data":"7eebae0b0db0ac4b03e40f4ffb0f3368042567cb7d80d84d6a31c9db2e8a30c8"} Oct 31 19:29:03.855973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.855740 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" event={"ID":"cb65a183-bd28-4a21-8895-82b7a39b4693","Type":"ContainerStarted","Data":"5d06458dd11bddf40af64d46b76348c5511f5f103928db85015ae94810300edc"} Oct 31 19:29:03.855973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.855791 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" event={"ID":"cb65a183-bd28-4a21-8895-82b7a39b4693","Type":"ContainerStarted","Data":"232453e97007bfbcb8416b1085daedfa3902cf855f54a7626364fa9f3201f04f"} Oct 31 19:29:03.884085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.883548 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" podStartSLOduration=1.883532225 podStartE2EDuration="1.883532225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:03.881664029 +0000 UTC m=+578.244484437" watchObservedRunningTime="2026-10-31 19:29:03.883532225 +0000 UTC m=+578.246352597" Oct 31 19:29:03.891470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.891442 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:03.910038 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.909978 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" podStartSLOduration=1.909960773 podStartE2EDuration="1.909960773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:03.907576936 +0000 UTC m=+578.270397309" watchObservedRunningTime="2026-10-31 19:29:03.909960773 +0000 UTC m=+578.272781146" Oct 31 19:29:03.932425 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:03.932365 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" podStartSLOduration=1.932349431 podStartE2EDuration="1.932349431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:03.931372211 +0000 UTC m=+578.294192583" watchObservedRunningTime="2026-10-31 19:29:03.932349431 +0000 UTC m=+578.295169804" Oct 31 19:29:04.231853 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:04.231813 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:04.237686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:04.237654 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:04.345131 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:04.345091 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:04.351678 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:04.351651 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:04.866650 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:04.866559 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:04.866650 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:04.866615 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:04.867528 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:04.867501 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:04.867795 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:04.867776 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:04.897347 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:04.897316 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:05.871604 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:05.871521 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:05.872303 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:05.872283 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:11.455060 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:11.455029 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq"] Oct 31 19:29:11.458761 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:11.458735 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq" Oct 31 19:29:11.472862 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:11.472834 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq"] Oct 31 19:29:11.530782 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:11.530745 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-8730d5-dockercfg-22bzb\"" Oct 31 19:29:11.533043 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:11.533023 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-8730d5\"" Oct 31 19:29:11.535578 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:11.535555 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d56bb36-ed0e-4127-9461-cbec40a46b95-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq\" (UID: \"7d56bb36-ed0e-4127-9461-cbec40a46b95\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq" Oct 31 19:29:11.535658 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:11.535635 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj5w6\" (UniqueName: \"kubernetes.io/projected/7d56bb36-ed0e-4127-9461-cbec40a46b95-kube-api-access-mj5w6\") pod \"payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq\" (UID: \"7d56bb36-ed0e-4127-9461-cbec40a46b95\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq" Oct 31 19:29:11.637035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:11.636993 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d56bb36-ed0e-4127-9461-cbec40a46b95-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq\" (UID: \"7d56bb36-ed0e-4127-9461-cbec40a46b95\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq" Oct 31 19:29:11.637222 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:11.637057 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj5w6\" (UniqueName: \"kubernetes.io/projected/7d56bb36-ed0e-4127-9461-cbec40a46b95-kube-api-access-mj5w6\") pod \"payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq\" (UID: \"7d56bb36-ed0e-4127-9461-cbec40a46b95\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq" Oct 31 19:29:11.638060 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:11.638023 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d56bb36-ed0e-4127-9461-cbec40a46b95-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq\" (UID: \"7d56bb36-ed0e-4127-9461-cbec40a46b95\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq" Oct 31 19:29:11.672851 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:11.672800 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj5w6\" (UniqueName: \"kubernetes.io/projected/7d56bb36-ed0e-4127-9461-cbec40a46b95-kube-api-access-mj5w6\") pod \"payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq\" (UID: \"7d56bb36-ed0e-4127-9461-cbec40a46b95\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq" Oct 31 19:29:11.779488 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:11.779399 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq" Oct 31 19:29:11.907335 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:11.907307 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9"] Oct 31 19:29:11.911404 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:11.911382 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9" Oct 31 19:29:11.926142 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:11.926117 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9"] Oct 31 19:29:11.994937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:11.994392 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-aeb292-dockercfg-rtxmc\"" Oct 31 19:29:11.994937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:11.994487 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-aeb292\"" Oct 31 19:29:12.051999 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:12.051717 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfx9l\" (UniqueName: \"kubernetes.io/projected/c34f06bd-3b50-4342-9719-bcccd4be955f-kube-api-access-gfx9l\") pod \"payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9\" (UID: \"c34f06bd-3b50-4342-9719-bcccd4be955f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9" Oct 31 19:29:12.051999 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:12.051796 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c34f06bd-3b50-4342-9719-bcccd4be955f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9\" (UID: \"c34f06bd-3b50-4342-9719-bcccd4be955f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9" Oct 31 19:29:12.078086 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:12.078058 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq"] Oct 31 19:29:12.079950 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:29:12.079902 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d56bb36_ed0e_4127_9461_cbec40a46b95.slice/crio-74df750d778ee9dc550afdf9e601fc8cc5a36668c0afbbaa38898ec35c14b3cd WatchSource:0}: Error finding container 74df750d778ee9dc550afdf9e601fc8cc5a36668c0afbbaa38898ec35c14b3cd: Status 404 returned error can't find the container with id 74df750d778ee9dc550afdf9e601fc8cc5a36668c0afbbaa38898ec35c14b3cd Oct 31 19:29:12.152684 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:12.152643 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c34f06bd-3b50-4342-9719-bcccd4be955f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9\" (UID: \"c34f06bd-3b50-4342-9719-bcccd4be955f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9" Oct 31 19:29:12.152857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:12.152837 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfx9l\" (UniqueName: \"kubernetes.io/projected/c34f06bd-3b50-4342-9719-bcccd4be955f-kube-api-access-gfx9l\") pod \"payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9\" (UID: \"c34f06bd-3b50-4342-9719-bcccd4be955f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9" Oct 31 19:29:12.153479 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:12.153458 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c34f06bd-3b50-4342-9719-bcccd4be955f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9\" (UID: \"c34f06bd-3b50-4342-9719-bcccd4be955f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9" Oct 31 19:29:12.184363 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:12.184331 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfx9l\" (UniqueName: \"kubernetes.io/projected/c34f06bd-3b50-4342-9719-bcccd4be955f-kube-api-access-gfx9l\") pod \"payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9\" (UID: \"c34f06bd-3b50-4342-9719-bcccd4be955f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9" Oct 31 19:29:12.236943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:12.235110 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9" Oct 31 19:29:12.456092 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:12.456056 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz"] Oct 31 19:29:12.460085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:12.460068 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz" Oct 31 19:29:12.469694 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:12.469671 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-23814d\"" Oct 31 19:29:12.472199 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:12.472179 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-23814d-dockercfg-j6tr8\"" Oct 31 19:29:12.476095 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:12.475733 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz"] Oct 31 19:29:12.529269 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:12.529234 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9"] Oct 31 19:29:12.548149 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:29:12.548113 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc34f06bd_3b50_4342_9719_bcccd4be955f.slice/crio-a26f04ac563f74403d5ad651d60849363345081672a2c68bf893b9197f292350 WatchSource:0}: Error finding container a26f04ac563f74403d5ad651d60849363345081672a2c68bf893b9197f292350: Status 404 returned error can't find the container with id a26f04ac563f74403d5ad651d60849363345081672a2c68bf893b9197f292350 Oct 31 19:29:12.557706 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:12.557675 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssdsx\" (UniqueName: \"kubernetes.io/projected/9a4949e1-4900-4ce8-9734-ce6e73bef410-kube-api-access-ssdsx\") pod \"payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz\" (UID: \"9a4949e1-4900-4ce8-9734-ce6e73bef410\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz" Oct 31 19:29:12.557812 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:12.557738 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a4949e1-4900-4ce8-9734-ce6e73bef410-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz\" (UID: \"9a4949e1-4900-4ce8-9734-ce6e73bef410\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz" Oct 31 19:29:12.663105 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:12.662503 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssdsx\" (UniqueName: \"kubernetes.io/projected/9a4949e1-4900-4ce8-9734-ce6e73bef410-kube-api-access-ssdsx\") pod \"payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz\" (UID: \"9a4949e1-4900-4ce8-9734-ce6e73bef410\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz" Oct 31 19:29:12.663105 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:12.662597 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a4949e1-4900-4ce8-9734-ce6e73bef410-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz\" (UID: \"9a4949e1-4900-4ce8-9734-ce6e73bef410\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz" Oct 31 19:29:12.663609 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:12.663582 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a4949e1-4900-4ce8-9734-ce6e73bef410-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz\" (UID: \"9a4949e1-4900-4ce8-9734-ce6e73bef410\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz" Oct 31 19:29:12.677289 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:12.677260 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssdsx\" (UniqueName: \"kubernetes.io/projected/9a4949e1-4900-4ce8-9734-ce6e73bef410-kube-api-access-ssdsx\") pod \"payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz\" (UID: \"9a4949e1-4900-4ce8-9734-ce6e73bef410\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz" Oct 31 19:29:12.715588 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:29:12.715513 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:29:12.774529 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:12.774431 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz" Oct 31 19:29:12.941933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:12.927594 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9" event={"ID":"c34f06bd-3b50-4342-9719-bcccd4be955f","Type":"ContainerStarted","Data":"a26f04ac563f74403d5ad651d60849363345081672a2c68bf893b9197f292350"} Oct 31 19:29:12.969488 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:12.969390 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq" event={"ID":"7d56bb36-ed0e-4127-9461-cbec40a46b95","Type":"ContainerStarted","Data":"74df750d778ee9dc550afdf9e601fc8cc5a36668c0afbbaa38898ec35c14b3cd"} Oct 31 19:29:13.154533 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:13.154325 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz"] Oct 31 19:29:13.160776 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:29:13.160747 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a4949e1_4900_4ce8_9734_ce6e73bef410.slice/crio-fea6cee7f9ca11971274a83c0b6743f4bb2bcf4175ef41eabb0cce56ddde57a6 WatchSource:0}: Error finding container fea6cee7f9ca11971274a83c0b6743f4bb2bcf4175ef41eabb0cce56ddde57a6: Status 404 returned error can't find the container with id fea6cee7f9ca11971274a83c0b6743f4bb2bcf4175ef41eabb0cce56ddde57a6 Oct 31 19:29:13.984273 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:13.984196 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz" event={"ID":"9a4949e1-4900-4ce8-9734-ce6e73bef410","Type":"ContainerStarted","Data":"fea6cee7f9ca11971274a83c0b6743f4bb2bcf4175ef41eabb0cce56ddde57a6"} Oct 31 19:29:15.393929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:15.393661 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt"] Oct 31 19:29:15.403158 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:15.403130 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt" Oct 31 19:29:15.490938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:15.455483 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt"] Oct 31 19:29:15.491223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:15.478150 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 19:29:15.491352 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:15.478266 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 19:29:15.491469 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:15.478341 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 19:29:15.491549 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:15.478387 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-23814d-serving-cert\"" Oct 31 19:29:15.491653 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:15.478478 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9dsr5\"" Oct 31 19:29:15.584034 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:15.583979 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20862e55-868f-40e8-af36-4f3e30cf9361-metrics-certs\") pod \"maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt\" (UID: \"20862e55-868f-40e8-af36-4f3e30cf9361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt" Oct 31 19:29:15.584034 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:15.584032 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj8lk\" (UniqueName: \"kubernetes.io/projected/20862e55-868f-40e8-af36-4f3e30cf9361-kube-api-access-rj8lk\") pod \"maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt\" (UID: \"20862e55-868f-40e8-af36-4f3e30cf9361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt" Oct 31 19:29:15.584314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:15.584068 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20862e55-868f-40e8-af36-4f3e30cf9361-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt\" (UID: \"20862e55-868f-40e8-af36-4f3e30cf9361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt" Oct 31 19:29:15.584314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:15.584130 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20862e55-868f-40e8-af36-4f3e30cf9361-maas-api-tls\") pod \"maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt\" (UID: \"20862e55-868f-40e8-af36-4f3e30cf9361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt" Oct 31 19:29:15.695032 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:15.690977 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20862e55-868f-40e8-af36-4f3e30cf9361-metrics-certs\") pod \"maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt\" (UID: \"20862e55-868f-40e8-af36-4f3e30cf9361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt" Oct 31 19:29:15.702953 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:15.699010 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj8lk\" (UniqueName: \"kubernetes.io/projected/20862e55-868f-40e8-af36-4f3e30cf9361-kube-api-access-rj8lk\") pod \"maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt\" (UID: \"20862e55-868f-40e8-af36-4f3e30cf9361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt" Oct 31 19:29:15.703250 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:15.703225 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20862e55-868f-40e8-af36-4f3e30cf9361-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt\" (UID: \"20862e55-868f-40e8-af36-4f3e30cf9361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt" Oct 31 19:29:15.711110 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:15.707691 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20862e55-868f-40e8-af36-4f3e30cf9361-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt\" (UID: \"20862e55-868f-40e8-af36-4f3e30cf9361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt" Oct 31 19:29:15.739934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:15.714117 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20862e55-868f-40e8-af36-4f3e30cf9361-maas-api-tls\") pod \"maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt\" (UID: \"20862e55-868f-40e8-af36-4f3e30cf9361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt" Oct 31 19:29:15.739934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:15.730656 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20862e55-868f-40e8-af36-4f3e30cf9361-metrics-certs\") pod \"maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt\" (UID: \"20862e55-868f-40e8-af36-4f3e30cf9361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt" Oct 31 19:29:15.767948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:15.757690 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj8lk\" (UniqueName: \"kubernetes.io/projected/20862e55-868f-40e8-af36-4f3e30cf9361-kube-api-access-rj8lk\") pod \"maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt\" (UID: \"20862e55-868f-40e8-af36-4f3e30cf9361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt" Oct 31 19:29:15.815051 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:15.814388 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20862e55-868f-40e8-af36-4f3e30cf9361-maas-api-tls\") pod \"maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt\" (UID: \"20862e55-868f-40e8-af36-4f3e30cf9361\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt" Oct 31 19:29:16.091982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:16.091944 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt" Oct 31 19:29:17.020466 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:17.020426 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt"] Oct 31 19:29:17.114018 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:17.113876 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt" event={"ID":"20862e55-868f-40e8-af36-4f3e30cf9361","Type":"ContainerStarted","Data":"8eeab951271e1db46e199f7bdff71cff017e037912fc399c5ee6d9a390068cde"} Oct 31 19:29:21.923122 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:21.923018 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:29:21.923122 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:21.923069 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" podUID="cb65a183-bd28-4a21-8895-82b7a39b4693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:29:22.366206 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:22.365543 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9" event={"ID":"c34f06bd-3b50-4342-9719-bcccd4be955f","Type":"ContainerStarted","Data":"938d28e68583a4153252ecef43fbb6e4c1e53561bcb89640f939bc7a91e6d39c"} Oct 31 19:29:22.366206 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:22.365625 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9" Oct 31 19:29:22.380848 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:22.380807 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq" event={"ID":"7d56bb36-ed0e-4127-9461-cbec40a46b95","Type":"ContainerStarted","Data":"b9e5a644cfb960de024dd921569b10f28f4c48cc6a8c226291b4a417e6ee3b35"} Oct 31 19:29:22.381571 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:22.381548 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq" Oct 31 19:29:22.473928 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:22.473831 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq" podStartSLOduration=1.772747538 podStartE2EDuration="11.473810844s" totalImagesPullingTime="9.701063306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:22.473319948 +0000 UTC m=+596.836140322" watchObservedRunningTime="2026-10-31 19:29:22.473810844 +0000 UTC m=+596.836631254" Oct 31 19:29:22.475217 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:22.475172 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9" podStartSLOduration=2.086396616 podStartE2EDuration="11.475158589s" totalImagesPullingTime="9.388761973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:22.406633912 +0000 UTC m=+596.769454284" watchObservedRunningTime="2026-10-31 19:29:22.475158589 +0000 UTC m=+596.837978962" Oct 31 19:29:23.392238 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.391397 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz" event={"ID":"9a4949e1-4900-4ce8-9734-ce6e73bef410","Type":"ContainerStarted","Data":"467a63236c40f6886d5f0c36f5079efa32cbc036ae0bdeefb18b4c0e4007267b"} Oct 31 19:29:23.392238 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.392151 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz" Oct 31 19:29:23.714839 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.714715 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz" podStartSLOduration=2.255698561 podStartE2EDuration="11.714693824s" totalImagesPullingTime="9.458995263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:23.431247986 +0000 UTC m=+597.794068359" watchObservedRunningTime="2026-10-31 19:29:23.714693824 +0000 UTC m=+598.077514199" Oct 31 19:29:23.717243 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.717208 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv"] Oct 31 19:29:23.744311 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.744281 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:29:23.747762 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.747662 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-home\") pod \"simulator-w4-996c32-kserve-d747cf979-rf6sv\" (UID: \"f2588d18-ac6b-403c-a350-11ecf531b468\") " pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:29:23.747762 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.747705 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2588d18-ac6b-403c-a350-11ecf531b468-tls-certs\") pod \"simulator-w4-996c32-kserve-d747cf979-rf6sv\" (UID: \"f2588d18-ac6b-403c-a350-11ecf531b468\") " pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:29:23.748437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.747734 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkck6\" (UniqueName: \"kubernetes.io/projected/f2588d18-ac6b-403c-a350-11ecf531b468-kube-api-access-wkck6\") pod \"simulator-w4-996c32-kserve-d747cf979-rf6sv\" (UID: \"f2588d18-ac6b-403c-a350-11ecf531b468\") " pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:29:23.748437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.748303 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-model-cache\") pod \"simulator-w4-996c32-kserve-d747cf979-rf6sv\" (UID: \"f2588d18-ac6b-403c-a350-11ecf531b468\") " pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:29:23.748437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.748372 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-tmp-dir\") pod \"simulator-w4-996c32-kserve-d747cf979-rf6sv\" (UID: \"f2588d18-ac6b-403c-a350-11ecf531b468\") " pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:29:23.748437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.748399 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-dshm\") pod \"simulator-w4-996c32-kserve-d747cf979-rf6sv\" (UID: \"f2588d18-ac6b-403c-a350-11ecf531b468\") " pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:29:23.750753 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.750728 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-996c32\"/\"openshift-service-ca.crt\"" Oct 31 19:29:23.751523 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.751502 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-996c32\"/\"kube-root-ca.crt\"" Oct 31 19:29:23.752871 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.752725 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-996c32\"/\"default-dockercfg-n5bxr\"" Oct 31 19:29:23.752871 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.752840 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-996c32\"/\"simulator-w4-996c32-kserve-self-signed-certs\"" Oct 31 19:29:23.765462 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.765432 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv"] Oct 31 19:29:23.849604 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.849563 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-home\") pod \"simulator-w4-996c32-kserve-d747cf979-rf6sv\" (UID: \"f2588d18-ac6b-403c-a350-11ecf531b468\") " pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:29:23.849770 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.849614 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2588d18-ac6b-403c-a350-11ecf531b468-tls-certs\") pod \"simulator-w4-996c32-kserve-d747cf979-rf6sv\" (UID: \"f2588d18-ac6b-403c-a350-11ecf531b468\") " pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:29:23.849770 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.849643 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkck6\" (UniqueName: \"kubernetes.io/projected/f2588d18-ac6b-403c-a350-11ecf531b468-kube-api-access-wkck6\") pod \"simulator-w4-996c32-kserve-d747cf979-rf6sv\" (UID: \"f2588d18-ac6b-403c-a350-11ecf531b468\") " pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:29:23.849770 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.849695 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-model-cache\") pod \"simulator-w4-996c32-kserve-d747cf979-rf6sv\" (UID: \"f2588d18-ac6b-403c-a350-11ecf531b468\") " pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:29:23.849770 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.849756 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-tmp-dir\") pod \"simulator-w4-996c32-kserve-d747cf979-rf6sv\" (UID: \"f2588d18-ac6b-403c-a350-11ecf531b468\") " pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:29:23.849900 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.849781 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-dshm\") pod \"simulator-w4-996c32-kserve-d747cf979-rf6sv\" (UID: \"f2588d18-ac6b-403c-a350-11ecf531b468\") " pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:29:23.850658 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.850299 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-home\") pod \"simulator-w4-996c32-kserve-d747cf979-rf6sv\" (UID: \"f2588d18-ac6b-403c-a350-11ecf531b468\") " pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:29:23.850658 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.850585 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-model-cache\") pod \"simulator-w4-996c32-kserve-d747cf979-rf6sv\" (UID: \"f2588d18-ac6b-403c-a350-11ecf531b468\") " pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:29:23.850658 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.850620 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-tmp-dir\") pod \"simulator-w4-996c32-kserve-d747cf979-rf6sv\" (UID: \"f2588d18-ac6b-403c-a350-11ecf531b468\") " pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:29:23.852538 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.852515 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-dshm\") pod \"simulator-w4-996c32-kserve-d747cf979-rf6sv\" (UID: \"f2588d18-ac6b-403c-a350-11ecf531b468\") " pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:29:23.853906 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.853882 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2588d18-ac6b-403c-a350-11ecf531b468-tls-certs\") pod \"simulator-w4-996c32-kserve-d747cf979-rf6sv\" (UID: \"f2588d18-ac6b-403c-a350-11ecf531b468\") " pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:29:23.867256 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:23.867222 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkck6\" (UniqueName: \"kubernetes.io/projected/f2588d18-ac6b-403c-a350-11ecf531b468-kube-api-access-wkck6\") pod \"simulator-w4-996c32-kserve-d747cf979-rf6sv\" (UID: \"f2588d18-ac6b-403c-a350-11ecf531b468\") " pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:29:24.093570 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:24.093522 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:29:24.454318 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:24.454081 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv"] Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.417350 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" event={"ID":"f2588d18-ac6b-403c-a350-11ecf531b468","Type":"ContainerStarted","Data":"8dd3210d637854ace4003879f1edf6cb2c24730adaf68c175add37586863e3bd"} Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.772613 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6"] Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.836291 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6"] Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.836396 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.847800 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8b083a\"/\"default-dockercfg-55sdv\"" Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.847878 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8b083a\"/\"simulator-w1-8b083a-kserve-self-signed-certs\"" Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.847976 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8b083a\"/\"openshift-service-ca.crt\"" Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.848056 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8b083a\"/\"kube-root-ca.crt\"" Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.879673 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cfae09f-0cde-4452-9ac1-babca8bb72a6-tls-certs\") pod \"simulator-w1-8b083a-kserve-699c484b58-5v9v6\" (UID: \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\") " pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.879705 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-dshm\") pod \"simulator-w1-8b083a-kserve-699c484b58-5v9v6\" (UID: \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\") " pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.879748 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-model-cache\") pod \"simulator-w1-8b083a-kserve-699c484b58-5v9v6\" (UID: \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\") " pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.879794 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx5kk\" (UniqueName: \"kubernetes.io/projected/0cfae09f-0cde-4452-9ac1-babca8bb72a6-kube-api-access-qx5kk\") pod \"simulator-w1-8b083a-kserve-699c484b58-5v9v6\" (UID: \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\") " pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.879835 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-tmp-dir\") pod \"simulator-w1-8b083a-kserve-699c484b58-5v9v6\" (UID: \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\") " pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.879849 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-home\") pod \"simulator-w1-8b083a-kserve-699c484b58-5v9v6\" (UID: \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\") " pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.918943 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.919574 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.980371 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-model-cache\") pod \"simulator-w1-8b083a-kserve-699c484b58-5v9v6\" (UID: \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\") " pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.980458 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx5kk\" (UniqueName: \"kubernetes.io/projected/0cfae09f-0cde-4452-9ac1-babca8bb72a6-kube-api-access-qx5kk\") pod \"simulator-w1-8b083a-kserve-699c484b58-5v9v6\" (UID: \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\") " pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.980536 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-tmp-dir\") pod \"simulator-w1-8b083a-kserve-699c484b58-5v9v6\" (UID: \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\") " pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.980563 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-home\") pod \"simulator-w1-8b083a-kserve-699c484b58-5v9v6\" (UID: \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\") " pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.980638 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cfae09f-0cde-4452-9ac1-babca8bb72a6-tls-certs\") pod \"simulator-w1-8b083a-kserve-699c484b58-5v9v6\" (UID: \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\") " pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.980664 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-dshm\") pod \"simulator-w1-8b083a-kserve-699c484b58-5v9v6\" (UID: \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\") " pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.980932 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-home\") pod \"simulator-w1-8b083a-kserve-699c484b58-5v9v6\" (UID: \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\") " pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.981050 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-model-cache\") pod \"simulator-w1-8b083a-kserve-699c484b58-5v9v6\" (UID: \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\") " pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.981053 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-tmp-dir\") pod \"simulator-w1-8b083a-kserve-699c484b58-5v9v6\" (UID: \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\") " pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.983213 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-dshm\") pod \"simulator-w1-8b083a-kserve-699c484b58-5v9v6\" (UID: \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\") " pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.983860 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cfae09f-0cde-4452-9ac1-babca8bb72a6-tls-certs\") pod \"simulator-w1-8b083a-kserve-699c484b58-5v9v6\" (UID: \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\") " pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:25.999068 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx5kk\" (UniqueName: \"kubernetes.io/projected/0cfae09f-0cde-4452-9ac1-babca8bb72a6-kube-api-access-qx5kk\") pod \"simulator-w1-8b083a-kserve-699c484b58-5v9v6\" (UID: \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\") " pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:29:26.276113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:26.203287 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:29:26.786100 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:26.426884 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt" event={"ID":"20862e55-868f-40e8-af36-4f3e30cf9361","Type":"ContainerStarted","Data":"3058a471f865d33201bb560a8b7a3031ebdee9a36c7f050af48f189b7992c3d1"} Oct 31 19:29:26.786100 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:26.427144 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt" Oct 31 19:29:26.786100 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:26.450420 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt" podStartSLOduration=3.201072566 podStartE2EDuration="11.450403397s" totalImagesPullingTime="8.249330831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:26.448860165 +0000 UTC m=+600.811680538" watchObservedRunningTime="2026-10-31 19:29:26.450403397 +0000 UTC m=+600.813223771" Oct 31 19:29:26.786100 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:26.511050 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6"] Oct 31 19:29:26.786100 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:29:26.512420 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cfae09f_0cde_4452_9ac1_babca8bb72a6.slice/crio-ff66440d61bfa84bb56fa794830450f285bdf43bf075ae0cdb836809ebe09155 WatchSource:0}: Error finding container ff66440d61bfa84bb56fa794830450f285bdf43bf075ae0cdb836809ebe09155: Status 404 returned error can't find the container with id ff66440d61bfa84bb56fa794830450f285bdf43bf075ae0cdb836809ebe09155 Oct 31 19:29:27.272278 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.272249 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp"] Oct 31 19:29:27.275815 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.275791 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:29:27.292751 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.292725 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-tmp-dir\") pod \"simulator-w0-23843f-kserve-6ffbc975c-mr8hp\" (UID: \"1a041bbc-a7be-4793-8981-b459959d8ed9\") " pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:29:27.293213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.292862 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-model-cache\") pod \"simulator-w0-23843f-kserve-6ffbc975c-mr8hp\" (UID: \"1a041bbc-a7be-4793-8981-b459959d8ed9\") " pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:29:27.293213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.292892 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-home\") pod \"simulator-w0-23843f-kserve-6ffbc975c-mr8hp\" (UID: \"1a041bbc-a7be-4793-8981-b459959d8ed9\") " pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:29:27.293213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.292990 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a041bbc-a7be-4793-8981-b459959d8ed9-tls-certs\") pod \"simulator-w0-23843f-kserve-6ffbc975c-mr8hp\" (UID: \"1a041bbc-a7be-4793-8981-b459959d8ed9\") " pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:29:27.293213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.293090 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-dshm\") pod \"simulator-w0-23843f-kserve-6ffbc975c-mr8hp\" (UID: \"1a041bbc-a7be-4793-8981-b459959d8ed9\") " pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:29:27.293213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.293138 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftvmj\" (UniqueName: \"kubernetes.io/projected/1a041bbc-a7be-4793-8981-b459959d8ed9-kube-api-access-ftvmj\") pod \"simulator-w0-23843f-kserve-6ffbc975c-mr8hp\" (UID: \"1a041bbc-a7be-4793-8981-b459959d8ed9\") " pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:29:27.298421 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.298362 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp"] Oct 31 19:29:27.301637 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.301617 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-23843f\"/\"simulator-w0-23843f-kserve-self-signed-certs\"" Oct 31 19:29:27.301637 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.301617 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-23843f\"/\"default-dockercfg-flgh4\"" Oct 31 19:29:27.321963 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.310545 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-23843f\"/\"openshift-service-ca.crt\"" Oct 31 19:29:27.321963 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.310591 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-23843f\"/\"kube-root-ca.crt\"" Oct 31 19:29:27.393732 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.393702 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-tmp-dir\") pod \"simulator-w0-23843f-kserve-6ffbc975c-mr8hp\" (UID: \"1a041bbc-a7be-4793-8981-b459959d8ed9\") " pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:29:27.393889 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.393788 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-model-cache\") pod \"simulator-w0-23843f-kserve-6ffbc975c-mr8hp\" (UID: \"1a041bbc-a7be-4793-8981-b459959d8ed9\") " pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:29:27.393889 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.393817 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-home\") pod \"simulator-w0-23843f-kserve-6ffbc975c-mr8hp\" (UID: \"1a041bbc-a7be-4793-8981-b459959d8ed9\") " pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:29:27.393889 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.393858 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a041bbc-a7be-4793-8981-b459959d8ed9-tls-certs\") pod \"simulator-w0-23843f-kserve-6ffbc975c-mr8hp\" (UID: \"1a041bbc-a7be-4793-8981-b459959d8ed9\") " pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:29:27.394076 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.393897 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-dshm\") pod \"simulator-w0-23843f-kserve-6ffbc975c-mr8hp\" (UID: \"1a041bbc-a7be-4793-8981-b459959d8ed9\") " pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:29:27.394076 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.393945 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftvmj\" (UniqueName: \"kubernetes.io/projected/1a041bbc-a7be-4793-8981-b459959d8ed9-kube-api-access-ftvmj\") pod \"simulator-w0-23843f-kserve-6ffbc975c-mr8hp\" (UID: \"1a041bbc-a7be-4793-8981-b459959d8ed9\") " pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:29:27.394281 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.394256 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-tmp-dir\") pod \"simulator-w0-23843f-kserve-6ffbc975c-mr8hp\" (UID: \"1a041bbc-a7be-4793-8981-b459959d8ed9\") " pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:29:27.394348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.394264 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-model-cache\") pod \"simulator-w0-23843f-kserve-6ffbc975c-mr8hp\" (UID: \"1a041bbc-a7be-4793-8981-b459959d8ed9\") " pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:29:27.394441 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.394421 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-home\") pod \"simulator-w0-23843f-kserve-6ffbc975c-mr8hp\" (UID: \"1a041bbc-a7be-4793-8981-b459959d8ed9\") " pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:29:27.396707 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.396676 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-dshm\") pod \"simulator-w0-23843f-kserve-6ffbc975c-mr8hp\" (UID: \"1a041bbc-a7be-4793-8981-b459959d8ed9\") " pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:29:27.396866 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.396847 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a041bbc-a7be-4793-8981-b459959d8ed9-tls-certs\") pod \"simulator-w0-23843f-kserve-6ffbc975c-mr8hp\" (UID: \"1a041bbc-a7be-4793-8981-b459959d8ed9\") " pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:29:27.416344 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.416313 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftvmj\" (UniqueName: \"kubernetes.io/projected/1a041bbc-a7be-4793-8981-b459959d8ed9-kube-api-access-ftvmj\") pod \"simulator-w0-23843f-kserve-6ffbc975c-mr8hp\" (UID: \"1a041bbc-a7be-4793-8981-b459959d8ed9\") " pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:29:27.434737 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.434701 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" event={"ID":"f2588d18-ac6b-403c-a350-11ecf531b468","Type":"ContainerStarted","Data":"672946e1083bcc7da2e7fbec1123b3ec89a7bad2324dd932b88861e0b95d0571"} Oct 31 19:29:27.436673 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.436651 4910 generic.go:356] "Generic (PLEG): container finished" podID="16761f60-662e-4eb1-a247-b00378149cc4" containerID="3333df1718d6f7235bfac8450c3c102e1e1de86bd13998a7f2a32a7cc5213b4d" exitCode=0 Oct 31 19:29:27.436783 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.436717 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" event={"ID":"16761f60-662e-4eb1-a247-b00378149cc4","Type":"ContainerDied","Data":"3333df1718d6f7235bfac8450c3c102e1e1de86bd13998a7f2a32a7cc5213b4d"} Oct 31 19:29:27.437112 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.437093 4910 scope.go:122] "RemoveContainer" containerID="3333df1718d6f7235bfac8450c3c102e1e1de86bd13998a7f2a32a7cc5213b4d" Oct 31 19:29:27.438381 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.438362 4910 generic.go:356] "Generic (PLEG): container finished" podID="204c09e1-cfce-4add-986d-165e78767839" containerID="de598823f128e0f7ff708756df6ed86a22abc66e59338acd2711304ab9b4a257" exitCode=0 Oct 31 19:29:27.438473 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.438400 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:29:27.438473 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.438423 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" event={"ID":"204c09e1-cfce-4add-986d-165e78767839","Type":"ContainerDied","Data":"de598823f128e0f7ff708756df6ed86a22abc66e59338acd2711304ab9b4a257"} Oct 31 19:29:27.438567 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.438469 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:29:27.438567 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.438512 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:29:27.438740 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.438722 4910 scope.go:122] "RemoveContainer" containerID="de598823f128e0f7ff708756df6ed86a22abc66e59338acd2711304ab9b4a257" Oct 31 19:29:27.439897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.439873 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:29:27.440014 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.439963 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:29:27.440066 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.440009 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:29:27.440462 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.440400 4910 generic.go:356] "Generic (PLEG): container finished" podID="cb65a183-bd28-4a21-8895-82b7a39b4693" containerID="5d06458dd11bddf40af64d46b76348c5511f5f103928db85015ae94810300edc" exitCode=0 Oct 31 19:29:27.440525 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.440471 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" event={"ID":"cb65a183-bd28-4a21-8895-82b7a39b4693","Type":"ContainerDied","Data":"5d06458dd11bddf40af64d46b76348c5511f5f103928db85015ae94810300edc"} Oct 31 19:29:27.440780 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.440761 4910 scope.go:122] "RemoveContainer" containerID="5d06458dd11bddf40af64d46b76348c5511f5f103928db85015ae94810300edc" Oct 31 19:29:27.442145 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.442053 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:29:27.442145 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.442126 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:29:27.442265 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.442173 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:29:27.442474 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.442454 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" event={"ID":"0cfae09f-0cde-4452-9ac1-babca8bb72a6","Type":"ContainerStarted","Data":"02b678ebf80d8fe3f9fe3af2cd19f227363957a991e1a4b930de18df07b19454"} Oct 31 19:29:27.442561 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.442486 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" event={"ID":"0cfae09f-0cde-4452-9ac1-babca8bb72a6","Type":"ContainerStarted","Data":"ff66440d61bfa84bb56fa794830450f285bdf43bf075ae0cdb836809ebe09155"} Oct 31 19:29:27.489490 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.489435 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podStartSLOduration=2.047682367 podStartE2EDuration="4.489418307s" totalImagesPullingTime="2.44173594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:27.488045259 +0000 UTC m=+601.850865630" watchObservedRunningTime="2026-10-31 19:29:27.489418307 +0000 UTC m=+601.852238682" Oct 31 19:29:27.685105 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.685026 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podStartSLOduration=2.145454859 podStartE2EDuration="2.685007506s" totalImagesPullingTime="539.552647ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:27.684102549 +0000 UTC m=+602.046922923" watchObservedRunningTime="2026-10-31 19:29:27.685007506 +0000 UTC m=+602.047827874" Oct 31 19:29:27.699923 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:27.699877 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:29:28.047992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:28.043134 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp"] Oct 31 19:29:28.094364 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:28.094333 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:29:28.204934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:28.204036 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:29:28.461858 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:28.461822 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" event={"ID":"1a041bbc-a7be-4793-8981-b459959d8ed9","Type":"ContainerStarted","Data":"714a92ed6901d5bb2cd9caa4b0622804552bee2d85d4f2a15cab515579fd9860"} Oct 31 19:29:28.475230 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:28.474221 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" event={"ID":"16761f60-662e-4eb1-a247-b00378149cc4","Type":"ContainerStarted","Data":"522881406a390a05402383a2d7dba0ae41adfe62527712ca787e5c2ecc06a5c5"} Oct 31 19:29:28.478708 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:28.478035 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" event={"ID":"204c09e1-cfce-4add-986d-165e78767839","Type":"ContainerStarted","Data":"88ba2c1333b4b3515b53ad6303bc2ea7ab6becea3f4431c72fd65d3472629b76"} Oct 31 19:29:28.486833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:28.486803 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" event={"ID":"cb65a183-bd28-4a21-8895-82b7a39b4693","Type":"ContainerStarted","Data":"0604db300e013ac905efb7c7adb034c94443496323246d49de6854ffd25fb06f"} Oct 31 19:29:28.900962 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:28.893351 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:28.900962 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:28.893474 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 31 19:29:28.900962 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:28.893518 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" podUID="cb65a183-bd28-4a21-8895-82b7a39b4693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 31 19:29:29.238469 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:29.238372 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:29.238842 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:29.238815 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 31 19:29:29.239036 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:29.239006 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" podUID="204c09e1-cfce-4add-986d-165e78767839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 31 19:29:29.346336 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:29.346299 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:29.347471 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:29.347371 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 31 19:29:29.347688 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:29.347664 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" podUID="16761f60-662e-4eb1-a247-b00378149cc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 31 19:29:29.525393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:29.517830 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" event={"ID":"1a041bbc-a7be-4793-8981-b459959d8ed9","Type":"ContainerStarted","Data":"1b4aace01da0e0dcf4737a6979ffe08853fa4872303bbbe236aca6871a50c637"} Oct 31 19:29:29.567073 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:29.567012 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podStartSLOduration=2.5669916219999998 podStartE2EDuration="2.566991622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:29.551479063 +0000 UTC m=+603.914299432" watchObservedRunningTime="2026-10-31 19:29:29.566991622 +0000 UTC m=+603.929811996" Oct 31 19:29:29.705096 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:29.705030 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:29:29.893561 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:29.893515 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 31 19:29:29.893760 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:29.893584 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" podUID="cb65a183-bd28-4a21-8895-82b7a39b4693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 31 19:29:30.236008 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:30.232347 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 31 19:29:30.236008 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:30.232410 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" podUID="204c09e1-cfce-4add-986d-165e78767839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 31 19:29:30.349429 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:30.349344 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 31 19:29:30.349429 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:30.349414 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" podUID="16761f60-662e-4eb1-a247-b00378149cc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 31 19:29:30.899935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:30.891367 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 31 19:29:30.899935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:30.891436 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" podUID="cb65a183-bd28-4a21-8895-82b7a39b4693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 31 19:29:31.242997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:31.235306 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 31 19:29:31.242997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:31.235369 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" podUID="204c09e1-cfce-4add-986d-165e78767839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 31 19:29:31.347944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:31.345304 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 31 19:29:31.347944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:31.345366 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" podUID="16761f60-662e-4eb1-a247-b00378149cc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 31 19:29:31.902367 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:31.902329 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 31 19:29:31.902992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:31.902948 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" podUID="cb65a183-bd28-4a21-8895-82b7a39b4693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 31 19:29:32.234134 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:32.233362 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 31 19:29:32.234134 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:32.233442 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" podUID="204c09e1-cfce-4add-986d-165e78767839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 31 19:29:32.349503 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:32.349285 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 31 19:29:32.349503 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:32.349348 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" podUID="16761f60-662e-4eb1-a247-b00378149cc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 31 19:29:32.475114 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:32.475081 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt" Oct 31 19:29:32.918076 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:32.918026 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 31 19:29:32.918633 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:32.918600 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" podUID="cb65a183-bd28-4a21-8895-82b7a39b4693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 31 19:29:33.234321 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:33.234231 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 31 19:29:33.234598 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:33.234562 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" podUID="204c09e1-cfce-4add-986d-165e78767839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 31 19:29:33.346340 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:33.346301 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Oct 31 19:29:33.347129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:33.347097 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" podUID="16761f60-662e-4eb1-a247-b00378149cc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Oct 31 19:29:33.407792 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:33.407548 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9" Oct 31 19:29:33.890900 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:33.890858 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 31 19:29:33.891117 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:33.890944 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" podUID="cb65a183-bd28-4a21-8895-82b7a39b4693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 31 19:29:34.094811 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:34.094767 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:29:34.116439 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:34.116400 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:29:34.232418 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:34.232333 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 31 19:29:34.232727 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:34.232699 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" podUID="204c09e1-cfce-4add-986d-165e78767839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 31 19:29:34.353763 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:34.352429 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:34.353763 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:34.353620 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:34.354470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:34.354452 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:29:34.415783 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:34.415755 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq" Oct 31 19:29:34.755949 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:34.755893 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:29:34.866778 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:34.866740 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:34.902247 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:34.902215 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:34.904113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:34.904089 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:34.906996 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:34.906970 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 19:29:35.237228 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:35.237196 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:35.421027 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:35.421000 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz" Oct 31 19:29:35.791945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:35.781603 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 19:29:36.203644 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:36.203607 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:29:36.217744 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:36.217707 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:29:36.785168 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:36.785136 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:29:37.700974 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:37.700936 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:29:37.714828 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:37.714793 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:29:37.793186 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:37.793156 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:29:42.717060 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:29:42.717012 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:29:42.909745 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:42.909710 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7"] Oct 31 19:29:42.913704 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:42.913685 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:29:42.921062 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:42.921035 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-8730d5\"/\"openshift-service-ca.crt\"" Oct 31 19:29:42.922234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:42.922213 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-8730d5\"/\"default-dockercfg-b5gbj\"" Oct 31 19:29:42.922443 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:42.922425 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-8730d5\"/\"kube-root-ca.crt\"" Oct 31 19:29:42.922505 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:42.922490 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-8730d5\"/\"simulator-w6-8730d5-kserve-self-signed-certs\"" Oct 31 19:29:42.939635 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:42.939609 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7"] Oct 31 19:29:42.993479 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:42.993377 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txh8f\" (UniqueName: \"kubernetes.io/projected/6db020e0-c316-47b9-ad20-a920d75f6138-kube-api-access-txh8f\") pod \"simulator-w6-8730d5-kserve-5754955c6b-5vxr7\" (UID: \"6db020e0-c316-47b9-ad20-a920d75f6138\") " pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:29:42.993642 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:42.993525 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-tmp-dir\") pod \"simulator-w6-8730d5-kserve-5754955c6b-5vxr7\" (UID: \"6db020e0-c316-47b9-ad20-a920d75f6138\") " pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:29:42.993642 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:42.993568 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db020e0-c316-47b9-ad20-a920d75f6138-tls-certs\") pod \"simulator-w6-8730d5-kserve-5754955c6b-5vxr7\" (UID: \"6db020e0-c316-47b9-ad20-a920d75f6138\") " pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:29:42.993753 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:42.993698 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-model-cache\") pod \"simulator-w6-8730d5-kserve-5754955c6b-5vxr7\" (UID: \"6db020e0-c316-47b9-ad20-a920d75f6138\") " pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:29:42.993837 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:42.993806 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-home\") pod \"simulator-w6-8730d5-kserve-5754955c6b-5vxr7\" (UID: \"6db020e0-c316-47b9-ad20-a920d75f6138\") " pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:29:42.994021 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:42.993870 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-dshm\") pod \"simulator-w6-8730d5-kserve-5754955c6b-5vxr7\" (UID: \"6db020e0-c316-47b9-ad20-a920d75f6138\") " pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:29:43.099530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:43.099408 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-model-cache\") pod \"simulator-w6-8730d5-kserve-5754955c6b-5vxr7\" (UID: \"6db020e0-c316-47b9-ad20-a920d75f6138\") " pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:29:43.099707 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:43.099579 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-home\") pod \"simulator-w6-8730d5-kserve-5754955c6b-5vxr7\" (UID: \"6db020e0-c316-47b9-ad20-a920d75f6138\") " pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:29:43.099707 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:43.099624 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-dshm\") pod \"simulator-w6-8730d5-kserve-5754955c6b-5vxr7\" (UID: \"6db020e0-c316-47b9-ad20-a920d75f6138\") " pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:29:43.099821 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:43.099754 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txh8f\" (UniqueName: \"kubernetes.io/projected/6db020e0-c316-47b9-ad20-a920d75f6138-kube-api-access-txh8f\") pod \"simulator-w6-8730d5-kserve-5754955c6b-5vxr7\" (UID: \"6db020e0-c316-47b9-ad20-a920d75f6138\") " pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:29:43.099884 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:43.099842 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-tmp-dir\") pod \"simulator-w6-8730d5-kserve-5754955c6b-5vxr7\" (UID: \"6db020e0-c316-47b9-ad20-a920d75f6138\") " pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:29:43.100085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:43.100053 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-model-cache\") pod \"simulator-w6-8730d5-kserve-5754955c6b-5vxr7\" (UID: \"6db020e0-c316-47b9-ad20-a920d75f6138\") " pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:29:43.100212 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:43.100123 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-home\") pod \"simulator-w6-8730d5-kserve-5754955c6b-5vxr7\" (UID: \"6db020e0-c316-47b9-ad20-a920d75f6138\") " pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:29:43.100274 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:43.100252 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db020e0-c316-47b9-ad20-a920d75f6138-tls-certs\") pod \"simulator-w6-8730d5-kserve-5754955c6b-5vxr7\" (UID: \"6db020e0-c316-47b9-ad20-a920d75f6138\") " pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:29:43.100431 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:43.100410 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-tmp-dir\") pod \"simulator-w6-8730d5-kserve-5754955c6b-5vxr7\" (UID: \"6db020e0-c316-47b9-ad20-a920d75f6138\") " pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:29:43.102643 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:43.102615 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-dshm\") pod \"simulator-w6-8730d5-kserve-5754955c6b-5vxr7\" (UID: \"6db020e0-c316-47b9-ad20-a920d75f6138\") " pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:29:43.103055 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:43.103029 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db020e0-c316-47b9-ad20-a920d75f6138-tls-certs\") pod \"simulator-w6-8730d5-kserve-5754955c6b-5vxr7\" (UID: \"6db020e0-c316-47b9-ad20-a920d75f6138\") " pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:29:43.112560 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:43.112539 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txh8f\" (UniqueName: \"kubernetes.io/projected/6db020e0-c316-47b9-ad20-a920d75f6138-kube-api-access-txh8f\") pod \"simulator-w6-8730d5-kserve-5754955c6b-5vxr7\" (UID: \"6db020e0-c316-47b9-ad20-a920d75f6138\") " pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:29:43.227091 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:43.227053 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:29:43.514699 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:43.514673 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7"] Oct 31 19:29:43.515648 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:29:43.515625 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6db020e0_c316_47b9_ad20_a920d75f6138.slice/crio-68ffe0ebacd6bbdaa05c32f94aec9f5a1d9a8497d384112309dfbe254ba71943 WatchSource:0}: Error finding container 68ffe0ebacd6bbdaa05c32f94aec9f5a1d9a8497d384112309dfbe254ba71943: Status 404 returned error can't find the container with id 68ffe0ebacd6bbdaa05c32f94aec9f5a1d9a8497d384112309dfbe254ba71943 Oct 31 19:29:43.830559 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:43.830519 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" event={"ID":"6db020e0-c316-47b9-ad20-a920d75f6138","Type":"ContainerStarted","Data":"c1eba0f875ec3982501b1251bc2c1fa1b6980d1ffe1cfb6e0b18def31e13e8de"} Oct 31 19:29:43.830559 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:43.830563 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" event={"ID":"6db020e0-c316-47b9-ad20-a920d75f6138","Type":"ContainerStarted","Data":"68ffe0ebacd6bbdaa05c32f94aec9f5a1d9a8497d384112309dfbe254ba71943"} Oct 31 19:29:43.852904 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:43.852844 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podStartSLOduration=1.8528245970000001 podStartE2EDuration="1.852824597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:43.849816255 +0000 UTC m=+618.212636629" watchObservedRunningTime="2026-10-31 19:29:43.852824597 +0000 UTC m=+618.215644970" Oct 31 19:29:44.227874 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:44.227787 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:29:50.714191 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:50.714151 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb"] Oct 31 19:29:50.719021 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:50.718994 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:29:50.730086 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:50.730059 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb"] Oct 31 19:29:50.730267 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:50.730251 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-23814d\"/\"openshift-service-ca.crt\"" Oct 31 19:29:50.730642 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:50.730626 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-23814d\"/\"kube-root-ca.crt\"" Oct 31 19:29:50.733125 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:50.733099 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-23814d\"/\"default-dockercfg-z7g59\"" Oct 31 19:29:50.733260 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:50.733139 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-23814d\"/\"simulator-w3-23814d-kserve-self-signed-certs\"" Oct 31 19:29:50.787727 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:50.787686 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-tmp-dir\") pod \"simulator-w3-23814d-kserve-b47579674-t5hnb\" (UID: \"423e19ac-6ae0-4240-9a04-17068d61ebd7\") " pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:29:50.787898 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:50.787767 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhdh2\" (UniqueName: \"kubernetes.io/projected/423e19ac-6ae0-4240-9a04-17068d61ebd7-kube-api-access-rhdh2\") pod \"simulator-w3-23814d-kserve-b47579674-t5hnb\" (UID: \"423e19ac-6ae0-4240-9a04-17068d61ebd7\") " pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:29:50.787898 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:50.787828 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-home\") pod \"simulator-w3-23814d-kserve-b47579674-t5hnb\" (UID: \"423e19ac-6ae0-4240-9a04-17068d61ebd7\") " pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:29:50.787898 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:50.787895 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-dshm\") pod \"simulator-w3-23814d-kserve-b47579674-t5hnb\" (UID: \"423e19ac-6ae0-4240-9a04-17068d61ebd7\") " pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:29:50.788543 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:50.787940 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-model-cache\") pod \"simulator-w3-23814d-kserve-b47579674-t5hnb\" (UID: \"423e19ac-6ae0-4240-9a04-17068d61ebd7\") " pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:29:50.788543 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:50.787978 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/423e19ac-6ae0-4240-9a04-17068d61ebd7-tls-certs\") pod \"simulator-w3-23814d-kserve-b47579674-t5hnb\" (UID: \"423e19ac-6ae0-4240-9a04-17068d61ebd7\") " pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:29:50.889414 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:50.889364 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-model-cache\") pod \"simulator-w3-23814d-kserve-b47579674-t5hnb\" (UID: \"423e19ac-6ae0-4240-9a04-17068d61ebd7\") " pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:29:50.889802 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:50.889767 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/423e19ac-6ae0-4240-9a04-17068d61ebd7-tls-certs\") pod \"simulator-w3-23814d-kserve-b47579674-t5hnb\" (UID: \"423e19ac-6ae0-4240-9a04-17068d61ebd7\") " pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:29:50.889980 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:50.889831 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-tmp-dir\") pod \"simulator-w3-23814d-kserve-b47579674-t5hnb\" (UID: \"423e19ac-6ae0-4240-9a04-17068d61ebd7\") " pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:29:50.889980 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:50.889902 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhdh2\" (UniqueName: \"kubernetes.io/projected/423e19ac-6ae0-4240-9a04-17068d61ebd7-kube-api-access-rhdh2\") pod \"simulator-w3-23814d-kserve-b47579674-t5hnb\" (UID: \"423e19ac-6ae0-4240-9a04-17068d61ebd7\") " pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:29:50.889980 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:50.889948 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-home\") pod \"simulator-w3-23814d-kserve-b47579674-t5hnb\" (UID: \"423e19ac-6ae0-4240-9a04-17068d61ebd7\") " pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:29:50.890163 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:50.890016 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-dshm\") pod \"simulator-w3-23814d-kserve-b47579674-t5hnb\" (UID: \"423e19ac-6ae0-4240-9a04-17068d61ebd7\") " pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:29:50.890796 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:50.890766 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-tmp-dir\") pod \"simulator-w3-23814d-kserve-b47579674-t5hnb\" (UID: \"423e19ac-6ae0-4240-9a04-17068d61ebd7\") " pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:29:50.894935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:50.891076 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-model-cache\") pod \"simulator-w3-23814d-kserve-b47579674-t5hnb\" (UID: \"423e19ac-6ae0-4240-9a04-17068d61ebd7\") " pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:29:50.909934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:50.896778 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-home\") pod \"simulator-w3-23814d-kserve-b47579674-t5hnb\" (UID: \"423e19ac-6ae0-4240-9a04-17068d61ebd7\") " pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:29:50.909934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:50.897559 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-dshm\") pod \"simulator-w3-23814d-kserve-b47579674-t5hnb\" (UID: \"423e19ac-6ae0-4240-9a04-17068d61ebd7\") " pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:29:50.920112 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:50.920001 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhdh2\" (UniqueName: \"kubernetes.io/projected/423e19ac-6ae0-4240-9a04-17068d61ebd7-kube-api-access-rhdh2\") pod \"simulator-w3-23814d-kserve-b47579674-t5hnb\" (UID: \"423e19ac-6ae0-4240-9a04-17068d61ebd7\") " pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:29:50.933694 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:50.933659 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/423e19ac-6ae0-4240-9a04-17068d61ebd7-tls-certs\") pod \"simulator-w3-23814d-kserve-b47579674-t5hnb\" (UID: \"423e19ac-6ae0-4240-9a04-17068d61ebd7\") " pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:29:51.034934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:51.034836 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:29:51.316449 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:51.316421 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb"] Oct 31 19:29:51.317156 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:29:51.317122 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod423e19ac_6ae0_4240_9a04_17068d61ebd7.slice/crio-dd63cc1040f37f3b138c3d78c3d56fb0d4b956fc25abef5a8664d2d617f68e6b WatchSource:0}: Error finding container dd63cc1040f37f3b138c3d78c3d56fb0d4b956fc25abef5a8664d2d617f68e6b: Status 404 returned error can't find the container with id dd63cc1040f37f3b138c3d78c3d56fb0d4b956fc25abef5a8664d2d617f68e6b Oct 31 19:29:51.898504 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:51.898468 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" event={"ID":"423e19ac-6ae0-4240-9a04-17068d61ebd7","Type":"ContainerStarted","Data":"2d20e17670a21aba5b9495e732fc2ad73f889bf5aa3342087a26713fd657e979"} Oct 31 19:29:51.899102 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:51.898510 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" event={"ID":"423e19ac-6ae0-4240-9a04-17068d61ebd7","Type":"ContainerStarted","Data":"dd63cc1040f37f3b138c3d78c3d56fb0d4b956fc25abef5a8664d2d617f68e6b"} Oct 31 19:29:51.920830 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:51.920777 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podStartSLOduration=1.920761575 podStartE2EDuration="1.920761575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:51.918709472 +0000 UTC m=+626.281529843" watchObservedRunningTime="2026-10-31 19:29:51.920761575 +0000 UTC m=+626.283581999" Oct 31 19:29:52.035520 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:52.035470 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:29:53.227379 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:53.227336 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:29:53.241095 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:53.241063 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:29:53.923835 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:53.923801 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:29:55.848898 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:55.848859 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp"] Oct 31 19:29:55.853539 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:55.853512 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:29:55.870306 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:55.870271 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp"] Oct 31 19:29:55.870570 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:55.870550 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8b083a\"/\"premium-w1-8b083a-kserve-self-signed-certs\"" Oct 31 19:29:55.949494 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:55.949449 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-dshm\") pod \"premium-w1-8b083a-kserve-f79655c75-5c7gp\" (UID: \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\") " pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:29:55.949701 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:55.949593 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8ghh\" (UniqueName: \"kubernetes.io/projected/d01fa9ef-252a-417c-b6d8-27375c4e1b90-kube-api-access-m8ghh\") pod \"premium-w1-8b083a-kserve-f79655c75-5c7gp\" (UID: \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\") " pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:29:55.949701 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:55.949691 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d01fa9ef-252a-417c-b6d8-27375c4e1b90-tls-certs\") pod \"premium-w1-8b083a-kserve-f79655c75-5c7gp\" (UID: \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\") " pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:29:55.949825 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:55.949765 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-tmp-dir\") pod \"premium-w1-8b083a-kserve-f79655c75-5c7gp\" (UID: \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\") " pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:29:55.949879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:55.949826 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-home\") pod \"premium-w1-8b083a-kserve-f79655c75-5c7gp\" (UID: \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\") " pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:29:55.949962 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:55.949941 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-model-cache\") pod \"premium-w1-8b083a-kserve-f79655c75-5c7gp\" (UID: \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\") " pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:29:56.050746 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:56.050704 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d01fa9ef-252a-417c-b6d8-27375c4e1b90-tls-certs\") pod \"premium-w1-8b083a-kserve-f79655c75-5c7gp\" (UID: \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\") " pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:29:56.050943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:56.050758 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-tmp-dir\") pod \"premium-w1-8b083a-kserve-f79655c75-5c7gp\" (UID: \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\") " pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:29:56.050943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:56.050819 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-home\") pod \"premium-w1-8b083a-kserve-f79655c75-5c7gp\" (UID: \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\") " pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:29:56.050943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:56.050862 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-model-cache\") pod \"premium-w1-8b083a-kserve-f79655c75-5c7gp\" (UID: \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\") " pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:29:56.051137 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:56.050948 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-dshm\") pod \"premium-w1-8b083a-kserve-f79655c75-5c7gp\" (UID: \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\") " pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:29:56.051137 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:56.051023 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ghh\" (UniqueName: \"kubernetes.io/projected/d01fa9ef-252a-417c-b6d8-27375c4e1b90-kube-api-access-m8ghh\") pod \"premium-w1-8b083a-kserve-f79655c75-5c7gp\" (UID: \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\") " pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:29:56.051389 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:56.051357 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-tmp-dir\") pod \"premium-w1-8b083a-kserve-f79655c75-5c7gp\" (UID: \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\") " pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:29:56.051389 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:56.051378 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-model-cache\") pod \"premium-w1-8b083a-kserve-f79655c75-5c7gp\" (UID: \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\") " pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:29:56.051724 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:56.051701 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-home\") pod \"premium-w1-8b083a-kserve-f79655c75-5c7gp\" (UID: \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\") " pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:29:56.053674 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:56.053644 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-dshm\") pod \"premium-w1-8b083a-kserve-f79655c75-5c7gp\" (UID: \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\") " pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:29:56.054438 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:56.054412 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d01fa9ef-252a-417c-b6d8-27375c4e1b90-tls-certs\") pod \"premium-w1-8b083a-kserve-f79655c75-5c7gp\" (UID: \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\") " pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:29:56.065072 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:56.065042 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8ghh\" (UniqueName: \"kubernetes.io/projected/d01fa9ef-252a-417c-b6d8-27375c4e1b90-kube-api-access-m8ghh\") pod \"premium-w1-8b083a-kserve-f79655c75-5c7gp\" (UID: \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\") " pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:29:56.175528 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:56.175438 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:29:56.459819 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:56.459789 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp"] Oct 31 19:29:56.461771 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:29:56.461731 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd01fa9ef_252a_417c_b6d8_27375c4e1b90.slice/crio-db44931ad4d177424c21bf97030ccd5c747df2a1705fbe9fcb9706447f5333ef WatchSource:0}: Error finding container db44931ad4d177424c21bf97030ccd5c747df2a1705fbe9fcb9706447f5333ef: Status 404 returned error can't find the container with id db44931ad4d177424c21bf97030ccd5c747df2a1705fbe9fcb9706447f5333ef Oct 31 19:29:56.936885 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:56.936846 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" event={"ID":"d01fa9ef-252a-417c-b6d8-27375c4e1b90","Type":"ContainerStarted","Data":"ff23d0413aa10b5e6e07e70fb0fda679a07d908a0c6fc9553919fa14cb065060"} Oct 31 19:29:56.937306 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:56.936897 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" event={"ID":"d01fa9ef-252a-417c-b6d8-27375c4e1b90","Type":"ContainerStarted","Data":"db44931ad4d177424c21bf97030ccd5c747df2a1705fbe9fcb9706447f5333ef"} Oct 31 19:29:56.963932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:56.960466 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podStartSLOduration=1.960433995 podStartE2EDuration="1.960433995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:56.95935077 +0000 UTC m=+631.322171143" watchObservedRunningTime="2026-10-31 19:29:56.960433995 +0000 UTC m=+631.323254367" Oct 31 19:29:57.175722 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:29:57.175689 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:30:00.155824 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.155786 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn"] Oct 31 19:30:00.162163 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.162140 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn" Oct 31 19:30:00.178404 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.174274 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj"] Oct 31 19:30:00.181671 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.181646 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:30:00.186591 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.185735 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68"] Oct 31 19:30:00.186591 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.186294 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj" Oct 31 19:30:00.195242 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.195217 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn"] Oct 31 19:30:00.195450 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.195434 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68"] Oct 31 19:30:00.195671 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.195653 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68" Oct 31 19:30:00.197043 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.197024 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj"] Oct 31 19:30:00.200786 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.200767 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8"] Oct 31 19:30:00.208153 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.208128 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8" Oct 31 19:30:00.217342 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.217313 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8"] Oct 31 19:30:00.301068 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.301034 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drx7t\" (UniqueName: \"kubernetes.io/projected/84fc6fdb-08c7-49bf-9fa0-84ead1928480-kube-api-access-drx7t\") pod \"maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn\" (UID: \"84fc6fdb-08c7-49bf-9fa0-84ead1928480\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn" Oct 31 19:30:00.301244 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.301165 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5939df20-88c8-41de-bc5d-2bcd988afe95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68\" (UID: \"5939df20-88c8-41de-bc5d-2bcd988afe95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68" Oct 31 19:30:00.301244 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.301235 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckxq4\" (UniqueName: \"kubernetes.io/projected/5939df20-88c8-41de-bc5d-2bcd988afe95-kube-api-access-ckxq4\") pod \"maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68\" (UID: \"5939df20-88c8-41de-bc5d-2bcd988afe95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68" Oct 31 19:30:00.301323 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.301266 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fc6fdb-08c7-49bf-9fa0-84ead1928480-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn\" (UID: \"84fc6fdb-08c7-49bf-9fa0-84ead1928480\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn" Oct 31 19:30:00.301323 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.301298 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87ea32e0-73b4-4d40-857e-e2940bff400c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8\" (UID: \"87ea32e0-73b4-4d40-857e-e2940bff400c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8" Oct 31 19:30:00.301407 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.301367 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbzsl\" (UniqueName: \"kubernetes.io/projected/3e0499ec-8fcf-4308-918f-f5ac8086860c-kube-api-access-dbzsl\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj\" (UID: \"3e0499ec-8fcf-4308-918f-f5ac8086860c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj" Oct 31 19:30:00.301441 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.301418 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e0499ec-8fcf-4308-918f-f5ac8086860c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj\" (UID: \"3e0499ec-8fcf-4308-918f-f5ac8086860c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj" Oct 31 19:30:00.301495 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.301479 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgp97\" (UniqueName: \"kubernetes.io/projected/87ea32e0-73b4-4d40-857e-e2940bff400c-kube-api-access-qgp97\") pod \"maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8\" (UID: \"87ea32e0-73b4-4d40-857e-e2940bff400c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8" Oct 31 19:30:00.402682 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.402644 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5939df20-88c8-41de-bc5d-2bcd988afe95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68\" (UID: \"5939df20-88c8-41de-bc5d-2bcd988afe95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68" Oct 31 19:30:00.402894 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.402696 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckxq4\" (UniqueName: \"kubernetes.io/projected/5939df20-88c8-41de-bc5d-2bcd988afe95-kube-api-access-ckxq4\") pod \"maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68\" (UID: \"5939df20-88c8-41de-bc5d-2bcd988afe95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68" Oct 31 19:30:00.402894 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.402716 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fc6fdb-08c7-49bf-9fa0-84ead1928480-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn\" (UID: \"84fc6fdb-08c7-49bf-9fa0-84ead1928480\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn" Oct 31 19:30:00.402894 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.402741 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87ea32e0-73b4-4d40-857e-e2940bff400c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8\" (UID: \"87ea32e0-73b4-4d40-857e-e2940bff400c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8" Oct 31 19:30:00.402894 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.402794 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbzsl\" (UniqueName: \"kubernetes.io/projected/3e0499ec-8fcf-4308-918f-f5ac8086860c-kube-api-access-dbzsl\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj\" (UID: \"3e0499ec-8fcf-4308-918f-f5ac8086860c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj" Oct 31 19:30:00.402894 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.402832 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e0499ec-8fcf-4308-918f-f5ac8086860c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj\" (UID: \"3e0499ec-8fcf-4308-918f-f5ac8086860c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj" Oct 31 19:30:00.402894 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.402893 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgp97\" (UniqueName: \"kubernetes.io/projected/87ea32e0-73b4-4d40-857e-e2940bff400c-kube-api-access-qgp97\") pod \"maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8\" (UID: \"87ea32e0-73b4-4d40-857e-e2940bff400c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8" Oct 31 19:30:00.403251 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.402972 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drx7t\" (UniqueName: \"kubernetes.io/projected/84fc6fdb-08c7-49bf-9fa0-84ead1928480-kube-api-access-drx7t\") pod \"maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn\" (UID: \"84fc6fdb-08c7-49bf-9fa0-84ead1928480\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn" Oct 31 19:30:00.403532 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.403506 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5939df20-88c8-41de-bc5d-2bcd988afe95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68\" (UID: \"5939df20-88c8-41de-bc5d-2bcd988afe95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68" Oct 31 19:30:00.403609 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.403514 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87ea32e0-73b4-4d40-857e-e2940bff400c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8\" (UID: \"87ea32e0-73b4-4d40-857e-e2940bff400c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8" Oct 31 19:30:00.403609 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.403551 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fc6fdb-08c7-49bf-9fa0-84ead1928480-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn\" (UID: \"84fc6fdb-08c7-49bf-9fa0-84ead1928480\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn" Oct 31 19:30:00.403609 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.403565 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e0499ec-8fcf-4308-918f-f5ac8086860c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj\" (UID: \"3e0499ec-8fcf-4308-918f-f5ac8086860c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj" Oct 31 19:30:00.416698 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.416635 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgp97\" (UniqueName: \"kubernetes.io/projected/87ea32e0-73b4-4d40-857e-e2940bff400c-kube-api-access-qgp97\") pod \"maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8\" (UID: \"87ea32e0-73b4-4d40-857e-e2940bff400c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8" Oct 31 19:30:00.417384 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.417360 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drx7t\" (UniqueName: \"kubernetes.io/projected/84fc6fdb-08c7-49bf-9fa0-84ead1928480-kube-api-access-drx7t\") pod \"maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn\" (UID: \"84fc6fdb-08c7-49bf-9fa0-84ead1928480\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn" Oct 31 19:30:00.419089 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.419064 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbzsl\" (UniqueName: \"kubernetes.io/projected/3e0499ec-8fcf-4308-918f-f5ac8086860c-kube-api-access-dbzsl\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj\" (UID: \"3e0499ec-8fcf-4308-918f-f5ac8086860c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj" Oct 31 19:30:00.421404 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.421379 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckxq4\" (UniqueName: \"kubernetes.io/projected/5939df20-88c8-41de-bc5d-2bcd988afe95-kube-api-access-ckxq4\") pod \"maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68\" (UID: \"5939df20-88c8-41de-bc5d-2bcd988afe95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68" Oct 31 19:30:00.501174 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.501136 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn" Oct 31 19:30:00.514656 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.514628 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj" Oct 31 19:30:00.523282 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.523250 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68" Oct 31 19:30:00.531392 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.531370 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8" Oct 31 19:30:00.815626 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:30:00.815594 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84fc6fdb_08c7_49bf_9fa0_84ead1928480.slice/crio-9cf22bb9dd819bd706c9ac1f704e646651bffda0ec040922a86d638457ec9ee5 WatchSource:0}: Error finding container 9cf22bb9dd819bd706c9ac1f704e646651bffda0ec040922a86d638457ec9ee5: Status 404 returned error can't find the container with id 9cf22bb9dd819bd706c9ac1f704e646651bffda0ec040922a86d638457ec9ee5 Oct 31 19:30:00.816409 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.816377 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn"] Oct 31 19:30:00.817826 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.817805 4910 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 19:30:00.818966 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:30:00.818798 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e0499ec_8fcf_4308_918f_f5ac8086860c.slice/crio-d77ef5d4b0c2ebc96bed5517ad73eb81edd9283df18d98385b13257576956958 WatchSource:0}: Error finding container d77ef5d4b0c2ebc96bed5517ad73eb81edd9283df18d98385b13257576956958: Status 404 returned error can't find the container with id d77ef5d4b0c2ebc96bed5517ad73eb81edd9283df18d98385b13257576956958 Oct 31 19:30:00.822344 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.822322 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj"] Oct 31 19:30:00.860971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.860809 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68"] Oct 31 19:30:00.864381 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:30:00.864353 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5939df20_88c8_41de_bc5d_2bcd988afe95.slice/crio-921a528994556f736b2fd7abcd5a0bb8d1a69e626f191dfffc4a901f3d0eccad WatchSource:0}: Error finding container 921a528994556f736b2fd7abcd5a0bb8d1a69e626f191dfffc4a901f3d0eccad: Status 404 returned error can't find the container with id 921a528994556f736b2fd7abcd5a0bb8d1a69e626f191dfffc4a901f3d0eccad Oct 31 19:30:00.864506 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.864490 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8"] Oct 31 19:30:00.868061 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:30:00.868041 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87ea32e0_73b4_4d40_857e_e2940bff400c.slice/crio-4027ebd48e749747d9412c4f18cab147c40af88b777ce269b135e3ad9718f70d WatchSource:0}: Error finding container 4027ebd48e749747d9412c4f18cab147c40af88b777ce269b135e3ad9718f70d: Status 404 returned error can't find the container with id 4027ebd48e749747d9412c4f18cab147c40af88b777ce269b135e3ad9718f70d Oct 31 19:30:00.964452 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.964352 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8" event={"ID":"87ea32e0-73b4-4d40-857e-e2940bff400c","Type":"ContainerStarted","Data":"4027ebd48e749747d9412c4f18cab147c40af88b777ce269b135e3ad9718f70d"} Oct 31 19:30:00.965790 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.965759 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn" event={"ID":"84fc6fdb-08c7-49bf-9fa0-84ead1928480","Type":"ContainerStarted","Data":"9cf22bb9dd819bd706c9ac1f704e646651bffda0ec040922a86d638457ec9ee5"} Oct 31 19:30:00.967478 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.967452 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68" event={"ID":"5939df20-88c8-41de-bc5d-2bcd988afe95","Type":"ContainerStarted","Data":"921a528994556f736b2fd7abcd5a0bb8d1a69e626f191dfffc4a901f3d0eccad"} Oct 31 19:30:00.968734 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:00.968710 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj" event={"ID":"3e0499ec-8fcf-4308-918f-f5ac8086860c","Type":"ContainerStarted","Data":"d77ef5d4b0c2ebc96bed5517ad73eb81edd9283df18d98385b13257576956958"} Oct 31 19:30:01.035235 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:01.035201 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:30:01.055040 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:01.055015 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:30:01.991540 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:01.991506 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:30:06.175887 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:06.175830 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:30:06.195420 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:06.195381 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:30:07.073000 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:07.072948 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8" event={"ID":"87ea32e0-73b4-4d40-857e-e2940bff400c","Type":"ContainerStarted","Data":"ecb233ff247a19227da6a95f9b01b6abab522c2e6814a8fb187ccddfb3a8d742"} Oct 31 19:30:07.074643 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:07.074616 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn" event={"ID":"84fc6fdb-08c7-49bf-9fa0-84ead1928480","Type":"ContainerStarted","Data":"df39eac7805c26a17a3811b5930809f6f14cd214c411d8aa37fc7d3eb85ca3e7"} Oct 31 19:30:07.076374 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:07.076350 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68" event={"ID":"5939df20-88c8-41de-bc5d-2bcd988afe95","Type":"ContainerStarted","Data":"4c6eb1c78647c19830e162ece47d291e8dfa4949682b757736d00e5a1d6baa0d"} Oct 31 19:30:07.077958 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:07.077933 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj" event={"ID":"3e0499ec-8fcf-4308-918f-f5ac8086860c","Type":"ContainerStarted","Data":"ddc1d2dd1efb3f5b1a98a26d325a578e6d1fb8a47054ee6281a9e912471b2a9c"} Oct 31 19:30:07.089271 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:07.089248 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:30:07.093516 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:07.093475 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8" podStartSLOduration=1.672834151 podStartE2EDuration="7.093465424s" totalImagesPullingTime="5.420631273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:07.091355046 +0000 UTC m=+641.454175420" watchObservedRunningTime="2026-10-31 19:30:07.093465424 +0000 UTC m=+641.456285821" Oct 31 19:30:07.112283 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:07.112239 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn" podStartSLOduration=1.548826412 podStartE2EDuration="7.112228802s" totalImagesPullingTime="5.56340239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:07.109344682 +0000 UTC m=+641.472165054" watchObservedRunningTime="2026-10-31 19:30:07.112228802 +0000 UTC m=+641.475049173" Oct 31 19:30:07.158277 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:07.158225 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj" podStartSLOduration=1.7348133909999999 podStartE2EDuration="7.158210528s" totalImagesPullingTime="5.423397137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:07.153140145 +0000 UTC m=+641.515960517" watchObservedRunningTime="2026-10-31 19:30:07.158210528 +0000 UTC m=+641.521030900" Oct 31 19:30:07.178247 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:07.178190 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68" podStartSLOduration=1.671906114 podStartE2EDuration="7.178173991s" totalImagesPullingTime="5.506267877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:07.176430505 +0000 UTC m=+641.539250870" watchObservedRunningTime="2026-10-31 19:30:07.178173991 +0000 UTC m=+641.540994364" Oct 31 19:30:08.091263 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:08.091228 4910 generic.go:356] "Generic (PLEG): container finished" podID="87ea32e0-73b4-4d40-857e-e2940bff400c" containerID="ecb233ff247a19227da6a95f9b01b6abab522c2e6814a8fb187ccddfb3a8d742" exitCode=60 Oct 31 19:30:08.091423 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:08.091313 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8" event={"ID":"87ea32e0-73b4-4d40-857e-e2940bff400c","Type":"ContainerDied","Data":"ecb233ff247a19227da6a95f9b01b6abab522c2e6814a8fb187ccddfb3a8d742"} Oct 31 19:30:08.091588 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:08.091569 4910 scope.go:122] "RemoveContainer" containerID="ecb233ff247a19227da6a95f9b01b6abab522c2e6814a8fb187ccddfb3a8d742" Oct 31 19:30:08.093046 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:08.093023 4910 generic.go:356] "Generic (PLEG): container finished" podID="84fc6fdb-08c7-49bf-9fa0-84ead1928480" containerID="df39eac7805c26a17a3811b5930809f6f14cd214c411d8aa37fc7d3eb85ca3e7" exitCode=60 Oct 31 19:30:08.093157 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:08.093111 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn" event={"ID":"84fc6fdb-08c7-49bf-9fa0-84ead1928480","Type":"ContainerDied","Data":"df39eac7805c26a17a3811b5930809f6f14cd214c411d8aa37fc7d3eb85ca3e7"} Oct 31 19:30:08.093311 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:08.093293 4910 scope.go:122] "RemoveContainer" containerID="df39eac7805c26a17a3811b5930809f6f14cd214c411d8aa37fc7d3eb85ca3e7" Oct 31 19:30:08.094820 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:08.094800 4910 generic.go:356] "Generic (PLEG): container finished" podID="5939df20-88c8-41de-bc5d-2bcd988afe95" containerID="4c6eb1c78647c19830e162ece47d291e8dfa4949682b757736d00e5a1d6baa0d" exitCode=60 Oct 31 19:30:08.094923 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:08.094880 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68" event={"ID":"5939df20-88c8-41de-bc5d-2bcd988afe95","Type":"ContainerDied","Data":"4c6eb1c78647c19830e162ece47d291e8dfa4949682b757736d00e5a1d6baa0d"} Oct 31 19:30:08.095084 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:08.095064 4910 scope.go:122] "RemoveContainer" containerID="4c6eb1c78647c19830e162ece47d291e8dfa4949682b757736d00e5a1d6baa0d" Oct 31 19:30:08.097147 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:08.097124 4910 generic.go:356] "Generic (PLEG): container finished" podID="3e0499ec-8fcf-4308-918f-f5ac8086860c" containerID="ddc1d2dd1efb3f5b1a98a26d325a578e6d1fb8a47054ee6281a9e912471b2a9c" exitCode=60 Oct 31 19:30:08.097258 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:08.097218 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj" event={"ID":"3e0499ec-8fcf-4308-918f-f5ac8086860c","Type":"ContainerDied","Data":"ddc1d2dd1efb3f5b1a98a26d325a578e6d1fb8a47054ee6281a9e912471b2a9c"} Oct 31 19:30:08.097461 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:08.097446 4910 scope.go:122] "RemoveContainer" containerID="ddc1d2dd1efb3f5b1a98a26d325a578e6d1fb8a47054ee6281a9e912471b2a9c" Oct 31 19:30:09.105421 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:09.105390 4910 generic.go:356] "Generic (PLEG): container finished" podID="3e0499ec-8fcf-4308-918f-f5ac8086860c" containerID="cc643fbff00b44cb473c591404e52aa9268ec185f65aca97c2aa813ed700342e" exitCode=60 Oct 31 19:30:09.105834 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:09.105437 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj" event={"ID":"3e0499ec-8fcf-4308-918f-f5ac8086860c","Type":"ContainerDied","Data":"cc643fbff00b44cb473c591404e52aa9268ec185f65aca97c2aa813ed700342e"} Oct 31 19:30:09.105834 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:09.105507 4910 scope.go:122] "RemoveContainer" containerID="ddc1d2dd1efb3f5b1a98a26d325a578e6d1fb8a47054ee6281a9e912471b2a9c" Oct 31 19:30:09.105993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:09.105973 4910 scope.go:122] "RemoveContainer" containerID="cc643fbff00b44cb473c591404e52aa9268ec185f65aca97c2aa813ed700342e" Oct 31 19:30:09.106306 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:30:09.106281 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj_odh-ai-gateway-infra(3e0499ec-8fcf-4308-918f-f5ac8086860c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj" podUID="3e0499ec-8fcf-4308-918f-f5ac8086860c" Oct 31 19:30:09.107849 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:09.107828 4910 generic.go:356] "Generic (PLEG): container finished" podID="87ea32e0-73b4-4d40-857e-e2940bff400c" containerID="1d79adca0791464bd155283857a9df78fe8e0831378eb3b750e217f83e06d306" exitCode=60 Oct 31 19:30:09.107969 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:09.107939 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8" event={"ID":"87ea32e0-73b4-4d40-857e-e2940bff400c","Type":"ContainerDied","Data":"1d79adca0791464bd155283857a9df78fe8e0831378eb3b750e217f83e06d306"} Oct 31 19:30:09.108254 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:09.108234 4910 scope.go:122] "RemoveContainer" containerID="1d79adca0791464bd155283857a9df78fe8e0831378eb3b750e217f83e06d306" Oct 31 19:30:09.108527 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:30:09.108505 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8_odh-ai-gateway-infra(87ea32e0-73b4-4d40-857e-e2940bff400c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8" podUID="87ea32e0-73b4-4d40-857e-e2940bff400c" Oct 31 19:30:09.110129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:09.110078 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn" event={"ID":"84fc6fdb-08c7-49bf-9fa0-84ead1928480","Type":"ContainerStarted","Data":"d4a89496430cb01abed7bf95a260d42523acc9b1843ff376bcc0823bf98307eb"} Oct 31 19:30:09.112124 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:09.112103 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68" event={"ID":"5939df20-88c8-41de-bc5d-2bcd988afe95","Type":"ContainerStarted","Data":"5ba49a30f0fd7bf0f1fc031bdf0a4342fee45d0d232a4f520f92410b9d71716f"} Oct 31 19:30:09.121044 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:09.121025 4910 scope.go:122] "RemoveContainer" containerID="ecb233ff247a19227da6a95f9b01b6abab522c2e6814a8fb187ccddfb3a8d742" Oct 31 19:30:10.122000 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:10.121968 4910 scope.go:122] "RemoveContainer" containerID="1d79adca0791464bd155283857a9df78fe8e0831378eb3b750e217f83e06d306" Oct 31 19:30:10.122463 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:30:10.122286 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8_odh-ai-gateway-infra(87ea32e0-73b4-4d40-857e-e2940bff400c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8" podUID="87ea32e0-73b4-4d40-857e-e2940bff400c" Oct 31 19:30:10.123833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:10.123798 4910 generic.go:356] "Generic (PLEG): container finished" podID="84fc6fdb-08c7-49bf-9fa0-84ead1928480" containerID="d4a89496430cb01abed7bf95a260d42523acc9b1843ff376bcc0823bf98307eb" exitCode=60 Oct 31 19:30:10.123999 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:10.123861 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn" event={"ID":"84fc6fdb-08c7-49bf-9fa0-84ead1928480","Type":"ContainerDied","Data":"d4a89496430cb01abed7bf95a260d42523acc9b1843ff376bcc0823bf98307eb"} Oct 31 19:30:10.124064 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:10.123999 4910 scope.go:122] "RemoveContainer" containerID="df39eac7805c26a17a3811b5930809f6f14cd214c411d8aa37fc7d3eb85ca3e7" Oct 31 19:30:10.124282 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:10.124248 4910 scope.go:122] "RemoveContainer" containerID="d4a89496430cb01abed7bf95a260d42523acc9b1843ff376bcc0823bf98307eb" Oct 31 19:30:10.124710 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:30:10.124682 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn_odh-ai-gateway-infra(84fc6fdb-08c7-49bf-9fa0-84ead1928480)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn" podUID="84fc6fdb-08c7-49bf-9fa0-84ead1928480" Oct 31 19:30:10.132594 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:10.132565 4910 generic.go:356] "Generic (PLEG): container finished" podID="5939df20-88c8-41de-bc5d-2bcd988afe95" containerID="5ba49a30f0fd7bf0f1fc031bdf0a4342fee45d0d232a4f520f92410b9d71716f" exitCode=60 Oct 31 19:30:10.132714 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:10.132597 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68" event={"ID":"5939df20-88c8-41de-bc5d-2bcd988afe95","Type":"ContainerDied","Data":"5ba49a30f0fd7bf0f1fc031bdf0a4342fee45d0d232a4f520f92410b9d71716f"} Oct 31 19:30:10.132880 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:10.132865 4910 scope.go:122] "RemoveContainer" containerID="5ba49a30f0fd7bf0f1fc031bdf0a4342fee45d0d232a4f520f92410b9d71716f" Oct 31 19:30:10.133139 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:30:10.133116 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68_odh-ai-gateway-infra(5939df20-88c8-41de-bc5d-2bcd988afe95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68" podUID="5939df20-88c8-41de-bc5d-2bcd988afe95" Oct 31 19:30:10.134580 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:10.134562 4910 scope.go:122] "RemoveContainer" containerID="cc643fbff00b44cb473c591404e52aa9268ec185f65aca97c2aa813ed700342e" Oct 31 19:30:10.134761 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:30:10.134743 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj_odh-ai-gateway-infra(3e0499ec-8fcf-4308-918f-f5ac8086860c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj" podUID="3e0499ec-8fcf-4308-918f-f5ac8086860c" Oct 31 19:30:10.143209 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:10.143190 4910 scope.go:122] "RemoveContainer" containerID="4c6eb1c78647c19830e162ece47d291e8dfa4949682b757736d00e5a1d6baa0d" Oct 31 19:30:11.144028 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:11.144002 4910 scope.go:122] "RemoveContainer" containerID="5ba49a30f0fd7bf0f1fc031bdf0a4342fee45d0d232a4f520f92410b9d71716f" Oct 31 19:30:11.144520 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:30:11.144309 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68_odh-ai-gateway-infra(5939df20-88c8-41de-bc5d-2bcd988afe95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68" podUID="5939df20-88c8-41de-bc5d-2bcd988afe95" Oct 31 19:30:11.146150 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:11.146130 4910 scope.go:122] "RemoveContainer" containerID="d4a89496430cb01abed7bf95a260d42523acc9b1843ff376bcc0823bf98307eb" Oct 31 19:30:11.146341 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:30:11.146321 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn_odh-ai-gateway-infra(84fc6fdb-08c7-49bf-9fa0-84ead1928480)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn" podUID="84fc6fdb-08c7-49bf-9fa0-84ead1928480" Oct 31 19:30:12.712779 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:30:12.712741 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:30:17.591256 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.591213 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc"] Oct 31 19:30:17.596681 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.596655 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.607751 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.607724 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-5bbf75-openshift-default-dockercfg-qv9k6\"" Oct 31 19:30:17.611378 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.611353 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc"] Oct 31 19:30:17.710894 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.710851 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bff40e4-ffc5-4104-8aed-4ff06167396c-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.710894 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.710895 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.711146 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.711025 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.711146 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.711053 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2k9w\" (UniqueName: \"kubernetes.io/projected/9bff40e4-ffc5-4104-8aed-4ff06167396c-kube-api-access-f2k9w\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.711249 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.711193 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.711318 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.711299 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.711360 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.711334 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.711415 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.711403 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.711457 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.711425 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.815097 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.813511 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bff40e4-ffc5-4104-8aed-4ff06167396c-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.815097 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.813580 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.815097 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.813623 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.815097 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.813658 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2k9w\" (UniqueName: \"kubernetes.io/projected/9bff40e4-ffc5-4104-8aed-4ff06167396c-kube-api-access-f2k9w\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.815097 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.813740 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.815097 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.813840 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.815097 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.813883 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.815097 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.813958 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.815097 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.813994 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.815864 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.815837 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bff40e4-ffc5-4104-8aed-4ff06167396c-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.817141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.816392 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.817141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.816685 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.818245 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.818215 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.818501 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.818480 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.829319 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.829281 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.833251 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.833206 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.838301 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.838256 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.839062 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.839039 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2k9w\" (UniqueName: \"kubernetes.io/projected/9bff40e4-ffc5-4104-8aed-4ff06167396c-kube-api-access-f2k9w\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:17.910840 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:17.910739 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:18.287305 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:18.287268 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc"] Oct 31 19:30:18.287481 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:30:18.287458 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bff40e4_ffc5_4104_8aed_4ff06167396c.slice/crio-615c4a375a89f426650624cb7f4629a6f0321a90f2d17222bc109ed0415fcb6b WatchSource:0}: Error finding container 615c4a375a89f426650624cb7f4629a6f0321a90f2d17222bc109ed0415fcb6b: Status 404 returned error can't find the container with id 615c4a375a89f426650624cb7f4629a6f0321a90f2d17222bc109ed0415fcb6b Oct 31 19:30:18.289735 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:18.289702 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:30:18.289825 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:18.289767 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:30:18.289825 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:18.289797 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:30:19.168979 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.168244 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz"] Oct 31 19:30:19.172476 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.172451 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:30:19.187938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.187280 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-aeb292\"/\"openshift-service-ca.crt\"" Oct 31 19:30:19.187938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.187507 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-aeb292\"/\"premium-w2-aeb292-kserve-self-signed-certs\"" Oct 31 19:30:19.187938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.187641 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-aeb292\"/\"default-dockercfg-g2p8n\"" Oct 31 19:30:19.187938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.187722 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-aeb292\"/\"kube-root-ca.crt\"" Oct 31 19:30:19.193025 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.193000 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz"] Oct 31 19:30:19.242935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.242741 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" event={"ID":"9bff40e4-ffc5-4104-8aed-4ff06167396c","Type":"ContainerStarted","Data":"22b01f8018ba6948f24c94d48d93246192f7e9fcd4f1aeb397b1994908b5d23f"} Oct 31 19:30:19.242935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.242832 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" event={"ID":"9bff40e4-ffc5-4104-8aed-4ff06167396c","Type":"ContainerStarted","Data":"615c4a375a89f426650624cb7f4629a6f0321a90f2d17222bc109ed0415fcb6b"} Oct 31 19:30:19.259935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.250233 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-model-cache\") pod \"premium-w2-aeb292-kserve-b5975b895-7pglz\" (UID: \"902b298d-646c-45e4-bff8-d505de51c689\") " pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:30:19.259935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.250284 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-dshm\") pod \"premium-w2-aeb292-kserve-b5975b895-7pglz\" (UID: \"902b298d-646c-45e4-bff8-d505de51c689\") " pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:30:19.259935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.250440 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slph9\" (UniqueName: \"kubernetes.io/projected/902b298d-646c-45e4-bff8-d505de51c689-kube-api-access-slph9\") pod \"premium-w2-aeb292-kserve-b5975b895-7pglz\" (UID: \"902b298d-646c-45e4-bff8-d505de51c689\") " pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:30:19.259935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.250494 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902b298d-646c-45e4-bff8-d505de51c689-tls-certs\") pod \"premium-w2-aeb292-kserve-b5975b895-7pglz\" (UID: \"902b298d-646c-45e4-bff8-d505de51c689\") " pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:30:19.259935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.250547 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-home\") pod \"premium-w2-aeb292-kserve-b5975b895-7pglz\" (UID: \"902b298d-646c-45e4-bff8-d505de51c689\") " pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:30:19.259935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.250577 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-tmp-dir\") pod \"premium-w2-aeb292-kserve-b5975b895-7pglz\" (UID: \"902b298d-646c-45e4-bff8-d505de51c689\") " pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:30:19.279224 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.279166 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" podStartSLOduration=2.279152105 podStartE2EDuration="2.279152105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:19.278131028 +0000 UTC m=+653.640951399" watchObservedRunningTime="2026-10-31 19:30:19.279152105 +0000 UTC m=+653.641972477" Oct 31 19:30:19.354933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.353979 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-home\") pod \"premium-w2-aeb292-kserve-b5975b895-7pglz\" (UID: \"902b298d-646c-45e4-bff8-d505de51c689\") " pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:30:19.354933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.354048 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-tmp-dir\") pod \"premium-w2-aeb292-kserve-b5975b895-7pglz\" (UID: \"902b298d-646c-45e4-bff8-d505de51c689\") " pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:30:19.354933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.354129 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-model-cache\") pod \"premium-w2-aeb292-kserve-b5975b895-7pglz\" (UID: \"902b298d-646c-45e4-bff8-d505de51c689\") " pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:30:19.354933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.354172 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-dshm\") pod \"premium-w2-aeb292-kserve-b5975b895-7pglz\" (UID: \"902b298d-646c-45e4-bff8-d505de51c689\") " pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:30:19.354933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.354289 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slph9\" (UniqueName: \"kubernetes.io/projected/902b298d-646c-45e4-bff8-d505de51c689-kube-api-access-slph9\") pod \"premium-w2-aeb292-kserve-b5975b895-7pglz\" (UID: \"902b298d-646c-45e4-bff8-d505de51c689\") " pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:30:19.354933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.354354 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902b298d-646c-45e4-bff8-d505de51c689-tls-certs\") pod \"premium-w2-aeb292-kserve-b5975b895-7pglz\" (UID: \"902b298d-646c-45e4-bff8-d505de51c689\") " pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:30:19.355565 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.355530 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-model-cache\") pod \"premium-w2-aeb292-kserve-b5975b895-7pglz\" (UID: \"902b298d-646c-45e4-bff8-d505de51c689\") " pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:30:19.375387 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.375355 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-home\") pod \"premium-w2-aeb292-kserve-b5975b895-7pglz\" (UID: \"902b298d-646c-45e4-bff8-d505de51c689\") " pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:30:19.379985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.379713 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902b298d-646c-45e4-bff8-d505de51c689-tls-certs\") pod \"premium-w2-aeb292-kserve-b5975b895-7pglz\" (UID: \"902b298d-646c-45e4-bff8-d505de51c689\") " pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:30:19.380170 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.380147 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-tmp-dir\") pod \"premium-w2-aeb292-kserve-b5975b895-7pglz\" (UID: \"902b298d-646c-45e4-bff8-d505de51c689\") " pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:30:19.389930 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.382686 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-dshm\") pod \"premium-w2-aeb292-kserve-b5975b895-7pglz\" (UID: \"902b298d-646c-45e4-bff8-d505de51c689\") " pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:30:19.405826 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.402581 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slph9\" (UniqueName: \"kubernetes.io/projected/902b298d-646c-45e4-bff8-d505de51c689-kube-api-access-slph9\") pod \"premium-w2-aeb292-kserve-b5975b895-7pglz\" (UID: \"902b298d-646c-45e4-bff8-d505de51c689\") " pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:30:19.492580 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.492492 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:30:19.752990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.751098 4910 scope.go:122] "RemoveContainer" containerID="d4a89496430cb01abed7bf95a260d42523acc9b1843ff376bcc0823bf98307eb" Oct 31 19:30:19.864663 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.864627 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz"] Oct 31 19:30:19.871156 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:30:19.871133 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod902b298d_646c_45e4_bff8_d505de51c689.slice/crio-c8b567521b56f9d30c61dd973f656359961d5d7e3ed251f5bb89cf43507cc848 WatchSource:0}: Error finding container c8b567521b56f9d30c61dd973f656359961d5d7e3ed251f5bb89cf43507cc848: Status 404 returned error can't find the container with id c8b567521b56f9d30c61dd973f656359961d5d7e3ed251f5bb89cf43507cc848 Oct 31 19:30:19.911679 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.911651 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:19.913017 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.912988 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 19:30:19.913106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:19.913038 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" podUID="9bff40e4-ffc5-4104-8aed-4ff06167396c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 19:30:20.306357 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:20.303837 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn" event={"ID":"84fc6fdb-08c7-49bf-9fa0-84ead1928480","Type":"ContainerStarted","Data":"7cf8fcd1a6d7551664d9626354ecf7a8a20bf520ec0580de939f8a1cc3083f82"} Oct 31 19:30:20.306357 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:20.305976 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" event={"ID":"902b298d-646c-45e4-bff8-d505de51c689","Type":"ContainerStarted","Data":"a8dca1b3e4cfbee24c6eafc0525b5ce3ca7cc2245527ea1bda7c9c4da4a06418"} Oct 31 19:30:20.306357 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:20.306016 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" event={"ID":"902b298d-646c-45e4-bff8-d505de51c689","Type":"ContainerStarted","Data":"c8b567521b56f9d30c61dd973f656359961d5d7e3ed251f5bb89cf43507cc848"} Oct 31 19:30:20.364770 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:20.364714 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podStartSLOduration=1.364699761 podStartE2EDuration="1.364699761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:20.362851087 +0000 UTC m=+654.725671460" watchObservedRunningTime="2026-10-31 19:30:20.364699761 +0000 UTC m=+654.727520133" Oct 31 19:30:20.495155 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:20.493363 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:30:20.913865 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:20.911268 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 19:30:20.913865 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:20.911328 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" podUID="9bff40e4-ffc5-4104-8aed-4ff06167396c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 19:30:21.342937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:21.342333 4910 generic.go:356] "Generic (PLEG): container finished" podID="84fc6fdb-08c7-49bf-9fa0-84ead1928480" containerID="7cf8fcd1a6d7551664d9626354ecf7a8a20bf520ec0580de939f8a1cc3083f82" exitCode=60 Oct 31 19:30:21.343440 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:21.343280 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn" event={"ID":"84fc6fdb-08c7-49bf-9fa0-84ead1928480","Type":"ContainerDied","Data":"7cf8fcd1a6d7551664d9626354ecf7a8a20bf520ec0580de939f8a1cc3083f82"} Oct 31 19:30:21.343440 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:21.343340 4910 scope.go:122] "RemoveContainer" containerID="d4a89496430cb01abed7bf95a260d42523acc9b1843ff376bcc0823bf98307eb" Oct 31 19:30:21.360935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:21.344284 4910 scope.go:122] "RemoveContainer" containerID="7cf8fcd1a6d7551664d9626354ecf7a8a20bf520ec0580de939f8a1cc3083f82" Oct 31 19:30:21.360935 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:30:21.344493 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn_odh-ai-gateway-infra(84fc6fdb-08c7-49bf-9fa0-84ead1928480)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn" podUID="84fc6fdb-08c7-49bf-9fa0-84ead1928480" Oct 31 19:30:21.394946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:21.390558 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn"] Oct 31 19:30:21.912051 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:21.912008 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 19:30:21.912260 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:21.912066 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" podUID="9bff40e4-ffc5-4104-8aed-4ff06167396c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 19:30:22.564601 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:22.564571 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn" Oct 31 19:30:22.620288 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:22.619304 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84fc6fdb-08c7-49bf-9fa0-84ead1928480-kube-api-access-drx7t\" (UniqueName: \"kubernetes.io/projected/84fc6fdb-08c7-49bf-9fa0-84ead1928480-kube-api-access-drx7t\") pod \"84fc6fdb-08c7-49bf-9fa0-84ead1928480\" (UID: \"84fc6fdb-08c7-49bf-9fa0-84ead1928480\") " Oct 31 19:30:22.620288 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:22.619365 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84fc6fdb-08c7-49bf-9fa0-84ead1928480-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fc6fdb-08c7-49bf-9fa0-84ead1928480-maas-api-service-ca\") pod \"84fc6fdb-08c7-49bf-9fa0-84ead1928480\" (UID: \"84fc6fdb-08c7-49bf-9fa0-84ead1928480\") " Oct 31 19:30:22.620989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:22.620658 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84fc6fdb-08c7-49bf-9fa0-84ead1928480-maas-api-service-ca" pod "84fc6fdb-08c7-49bf-9fa0-84ead1928480" (UID: "84fc6fdb-08c7-49bf-9fa0-84ead1928480"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:30:22.639118 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:22.639064 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84fc6fdb-08c7-49bf-9fa0-84ead1928480-kube-api-access-drx7t" pod "84fc6fdb-08c7-49bf-9fa0-84ead1928480" (UID: "84fc6fdb-08c7-49bf-9fa0-84ead1928480"). InnerVolumeSpecName "kube-api-access-drx7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:30:22.722937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:22.722792 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drx7t\" (UniqueName: \"kubernetes.io/projected/84fc6fdb-08c7-49bf-9fa0-84ead1928480-kube-api-access-drx7t\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:30:22.722937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:22.722853 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84fc6fdb-08c7-49bf-9fa0-84ead1928480-maas-api-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:30:22.749868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:22.747976 4910 scope.go:122] "RemoveContainer" containerID="1d79adca0791464bd155283857a9df78fe8e0831378eb3b750e217f83e06d306" Oct 31 19:30:22.912010 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:22.911894 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 19:30:22.912010 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:22.911970 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" podUID="9bff40e4-ffc5-4104-8aed-4ff06167396c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 19:30:23.374589 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:23.374552 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn" Oct 31 19:30:23.374589 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:23.374551 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn" event={"ID":"84fc6fdb-08c7-49bf-9fa0-84ead1928480","Type":"ContainerDied","Data":"9cf22bb9dd819bd706c9ac1f704e646651bffda0ec040922a86d638457ec9ee5"} Oct 31 19:30:23.374846 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:23.374693 4910 scope.go:122] "RemoveContainer" containerID="7cf8fcd1a6d7551664d9626354ecf7a8a20bf520ec0580de939f8a1cc3083f82" Oct 31 19:30:23.377530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:23.377505 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8" event={"ID":"87ea32e0-73b4-4d40-857e-e2940bff400c","Type":"ContainerStarted","Data":"8beaeb3a3023e8dbde44372383f211af0b74a35f2d55d9870f46476a827b8505"} Oct 31 19:30:23.449373 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:23.444516 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn"] Oct 31 19:30:23.460934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:23.460629 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891250-m2qdn"] Oct 31 19:30:23.753948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:23.753354 4910 scope.go:122] "RemoveContainer" containerID="cc643fbff00b44cb473c591404e52aa9268ec185f65aca97c2aa813ed700342e" Oct 31 19:30:23.761146 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:23.761123 4910 scope.go:122] "RemoveContainer" containerID="5ba49a30f0fd7bf0f1fc031bdf0a4342fee45d0d232a4f520f92410b9d71716f" Oct 31 19:30:23.772944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:23.772678 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84fc6fdb-08c7-49bf-9fa0-84ead1928480" path="/var/lib/kubelet/pods/84fc6fdb-08c7-49bf-9fa0-84ead1928480/volumes" Oct 31 19:30:23.912080 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:23.912041 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 31 19:30:23.912291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:23.912104 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" podUID="9bff40e4-ffc5-4104-8aed-4ff06167396c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 31 19:30:24.386740 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:24.386709 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj" event={"ID":"3e0499ec-8fcf-4308-918f-f5ac8086860c","Type":"ContainerStarted","Data":"207c89e8eda64722e9324edd67fb82a2d83e615da8c497b2f9e2d4fb7361aa24"} Oct 31 19:30:24.389847 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:24.389820 4910 generic.go:356] "Generic (PLEG): container finished" podID="87ea32e0-73b4-4d40-857e-e2940bff400c" containerID="8beaeb3a3023e8dbde44372383f211af0b74a35f2d55d9870f46476a827b8505" exitCode=60 Oct 31 19:30:24.390046 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:24.389949 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8" event={"ID":"87ea32e0-73b4-4d40-857e-e2940bff400c","Type":"ContainerDied","Data":"8beaeb3a3023e8dbde44372383f211af0b74a35f2d55d9870f46476a827b8505"} Oct 31 19:30:24.390046 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:24.390003 4910 scope.go:122] "RemoveContainer" containerID="1d79adca0791464bd155283857a9df78fe8e0831378eb3b750e217f83e06d306" Oct 31 19:30:24.390359 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:24.390340 4910 scope.go:122] "RemoveContainer" containerID="8beaeb3a3023e8dbde44372383f211af0b74a35f2d55d9870f46476a827b8505" Oct 31 19:30:24.390602 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:30:24.390580 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8_odh-ai-gateway-infra(87ea32e0-73b4-4d40-857e-e2940bff400c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8" podUID="87ea32e0-73b4-4d40-857e-e2940bff400c" Oct 31 19:30:24.394835 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:24.394636 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68" event={"ID":"5939df20-88c8-41de-bc5d-2bcd988afe95","Type":"ContainerStarted","Data":"b8f7f7e35b943e3490cdf2cc3b01773a89f8514f481d2e120b852a19b2b5ba82"} Oct 31 19:30:24.417636 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:24.417609 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8"] Oct 31 19:30:25.406807 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:25.406766 4910 generic.go:356] "Generic (PLEG): container finished" podID="5939df20-88c8-41de-bc5d-2bcd988afe95" containerID="b8f7f7e35b943e3490cdf2cc3b01773a89f8514f481d2e120b852a19b2b5ba82" exitCode=60 Oct 31 19:30:25.407283 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:25.406896 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68" event={"ID":"5939df20-88c8-41de-bc5d-2bcd988afe95","Type":"ContainerDied","Data":"b8f7f7e35b943e3490cdf2cc3b01773a89f8514f481d2e120b852a19b2b5ba82"} Oct 31 19:30:25.407283 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:25.407025 4910 scope.go:122] "RemoveContainer" containerID="5ba49a30f0fd7bf0f1fc031bdf0a4342fee45d0d232a4f520f92410b9d71716f" Oct 31 19:30:25.407560 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:25.407434 4910 scope.go:122] "RemoveContainer" containerID="b8f7f7e35b943e3490cdf2cc3b01773a89f8514f481d2e120b852a19b2b5ba82" Oct 31 19:30:25.407796 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:30:25.407772 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68_odh-ai-gateway-infra(5939df20-88c8-41de-bc5d-2bcd988afe95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68" podUID="5939df20-88c8-41de-bc5d-2bcd988afe95" Oct 31 19:30:25.411653 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:25.410811 4910 generic.go:356] "Generic (PLEG): container finished" podID="3e0499ec-8fcf-4308-918f-f5ac8086860c" containerID="207c89e8eda64722e9324edd67fb82a2d83e615da8c497b2f9e2d4fb7361aa24" exitCode=60 Oct 31 19:30:25.411653 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:25.410945 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj" event={"ID":"3e0499ec-8fcf-4308-918f-f5ac8086860c","Type":"ContainerDied","Data":"207c89e8eda64722e9324edd67fb82a2d83e615da8c497b2f9e2d4fb7361aa24"} Oct 31 19:30:25.411653 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:25.411337 4910 scope.go:122] "RemoveContainer" containerID="207c89e8eda64722e9324edd67fb82a2d83e615da8c497b2f9e2d4fb7361aa24" Oct 31 19:30:25.411653 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:30:25.411631 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj_odh-ai-gateway-infra(3e0499ec-8fcf-4308-918f-f5ac8086860c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj" podUID="3e0499ec-8fcf-4308-918f-f5ac8086860c" Oct 31 19:30:25.422831 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:25.422796 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj"] Oct 31 19:30:25.430927 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:25.430887 4910 scope.go:122] "RemoveContainer" containerID="cc643fbff00b44cb473c591404e52aa9268ec185f65aca97c2aa813ed700342e" Oct 31 19:30:25.453137 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:25.453113 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68"] Oct 31 19:30:25.613221 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:25.613194 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8" Oct 31 19:30:25.691686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:25.690305 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87ea32e0-73b4-4d40-857e-e2940bff400c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87ea32e0-73b4-4d40-857e-e2940bff400c-maas-api-service-ca\") pod \"87ea32e0-73b4-4d40-857e-e2940bff400c\" (UID: \"87ea32e0-73b4-4d40-857e-e2940bff400c\") " Oct 31 19:30:25.691686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:25.690521 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87ea32e0-73b4-4d40-857e-e2940bff400c-kube-api-access-qgp97\" (UniqueName: \"kubernetes.io/projected/87ea32e0-73b4-4d40-857e-e2940bff400c-kube-api-access-qgp97\") pod \"87ea32e0-73b4-4d40-857e-e2940bff400c\" (UID: \"87ea32e0-73b4-4d40-857e-e2940bff400c\") " Oct 31 19:30:25.695945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:25.695884 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87ea32e0-73b4-4d40-857e-e2940bff400c-maas-api-service-ca" pod "87ea32e0-73b4-4d40-857e-e2940bff400c" (UID: "87ea32e0-73b4-4d40-857e-e2940bff400c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:30:25.698316 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:25.697259 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87ea32e0-73b4-4d40-857e-e2940bff400c-kube-api-access-qgp97" pod "87ea32e0-73b4-4d40-857e-e2940bff400c" (UID: "87ea32e0-73b4-4d40-857e-e2940bff400c"). InnerVolumeSpecName "kube-api-access-qgp97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:30:25.793029 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:25.792999 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87ea32e0-73b4-4d40-857e-e2940bff400c-maas-api-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:30:25.793029 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:25.793023 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgp97\" (UniqueName: \"kubernetes.io/projected/87ea32e0-73b4-4d40-857e-e2940bff400c-kube-api-access-qgp97\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:30:25.833153 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:25.833118 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:25.833959 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:25.833939 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:25.834607 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:25.834580 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:30:26.420753 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:26.420716 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8" Oct 31 19:30:26.421189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:26.420752 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8" event={"ID":"87ea32e0-73b4-4d40-857e-e2940bff400c","Type":"ContainerDied","Data":"4027ebd48e749747d9412c4f18cab147c40af88b777ce269b135e3ad9718f70d"} Oct 31 19:30:26.421189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:26.420863 4910 scope.go:122] "RemoveContainer" containerID="8beaeb3a3023e8dbde44372383f211af0b74a35f2d55d9870f46476a827b8505" Oct 31 19:30:26.491012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:26.490978 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8"] Oct 31 19:30:26.494445 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:26.494399 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891250-s6wp8"] Oct 31 19:30:26.618696 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:26.618673 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj" Oct 31 19:30:26.621712 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:26.621693 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68" Oct 31 19:30:26.704262 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:26.704175 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e0499ec-8fcf-4308-918f-f5ac8086860c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e0499ec-8fcf-4308-918f-f5ac8086860c-maas-api-service-ca\") pod \"3e0499ec-8fcf-4308-918f-f5ac8086860c\" (UID: \"3e0499ec-8fcf-4308-918f-f5ac8086860c\") " Oct 31 19:30:26.704432 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:26.704317 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5939df20-88c8-41de-bc5d-2bcd988afe95-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5939df20-88c8-41de-bc5d-2bcd988afe95-maas-api-service-ca\") pod \"5939df20-88c8-41de-bc5d-2bcd988afe95\" (UID: \"5939df20-88c8-41de-bc5d-2bcd988afe95\") " Oct 31 19:30:26.704432 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:26.704410 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5939df20-88c8-41de-bc5d-2bcd988afe95-kube-api-access-ckxq4\" (UniqueName: \"kubernetes.io/projected/5939df20-88c8-41de-bc5d-2bcd988afe95-kube-api-access-ckxq4\") pod \"5939df20-88c8-41de-bc5d-2bcd988afe95\" (UID: \"5939df20-88c8-41de-bc5d-2bcd988afe95\") " Oct 31 19:30:26.704534 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:26.704496 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e0499ec-8fcf-4308-918f-f5ac8086860c-kube-api-access-dbzsl\" (UniqueName: \"kubernetes.io/projected/3e0499ec-8fcf-4308-918f-f5ac8086860c-kube-api-access-dbzsl\") pod \"3e0499ec-8fcf-4308-918f-f5ac8086860c\" (UID: \"3e0499ec-8fcf-4308-918f-f5ac8086860c\") " Oct 31 19:30:26.704641 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:26.704612 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e0499ec-8fcf-4308-918f-f5ac8086860c-maas-api-service-ca" pod "3e0499ec-8fcf-4308-918f-f5ac8086860c" (UID: "3e0499ec-8fcf-4308-918f-f5ac8086860c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:30:26.704712 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:26.704687 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5939df20-88c8-41de-bc5d-2bcd988afe95-maas-api-service-ca" pod "5939df20-88c8-41de-bc5d-2bcd988afe95" (UID: "5939df20-88c8-41de-bc5d-2bcd988afe95"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:30:26.704894 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:26.704878 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5939df20-88c8-41de-bc5d-2bcd988afe95-maas-api-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:30:26.705011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:26.704899 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e0499ec-8fcf-4308-918f-f5ac8086860c-maas-api-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:30:26.706794 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:26.706764 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e0499ec-8fcf-4308-918f-f5ac8086860c-kube-api-access-dbzsl" pod "3e0499ec-8fcf-4308-918f-f5ac8086860c" (UID: "3e0499ec-8fcf-4308-918f-f5ac8086860c"). InnerVolumeSpecName "kube-api-access-dbzsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:30:26.706975 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:26.706955 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5939df20-88c8-41de-bc5d-2bcd988afe95-kube-api-access-ckxq4" pod "5939df20-88c8-41de-bc5d-2bcd988afe95" (UID: "5939df20-88c8-41de-bc5d-2bcd988afe95"). InnerVolumeSpecName "kube-api-access-ckxq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:30:26.806081 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:26.806037 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckxq4\" (UniqueName: \"kubernetes.io/projected/5939df20-88c8-41de-bc5d-2bcd988afe95-kube-api-access-ckxq4\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:30:26.806081 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:26.806074 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbzsl\" (UniqueName: \"kubernetes.io/projected/3e0499ec-8fcf-4308-918f-f5ac8086860c-kube-api-access-dbzsl\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:30:27.432006 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:27.431973 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68" Oct 31 19:30:27.432492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:27.432032 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68" event={"ID":"5939df20-88c8-41de-bc5d-2bcd988afe95","Type":"ContainerDied","Data":"921a528994556f736b2fd7abcd5a0bb8d1a69e626f191dfffc4a901f3d0eccad"} Oct 31 19:30:27.432492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:27.432096 4910 scope.go:122] "RemoveContainer" containerID="b8f7f7e35b943e3490cdf2cc3b01773a89f8514f481d2e120b852a19b2b5ba82" Oct 31 19:30:27.433604 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:27.433582 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj" Oct 31 19:30:27.433724 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:27.433580 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj" event={"ID":"3e0499ec-8fcf-4308-918f-f5ac8086860c","Type":"ContainerDied","Data":"d77ef5d4b0c2ebc96bed5517ad73eb81edd9283df18d98385b13257576956958"} Oct 31 19:30:27.444229 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:27.444160 4910 scope.go:122] "RemoveContainer" containerID="207c89e8eda64722e9324edd67fb82a2d83e615da8c497b2f9e2d4fb7361aa24" Oct 31 19:30:27.463755 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:27.463725 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68"] Oct 31 19:30:27.470590 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:27.470560 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891250-rsm68"] Oct 31 19:30:27.484818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:27.484790 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj"] Oct 31 19:30:27.492292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:27.492265 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891250-5p6pj"] Oct 31 19:30:27.755209 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:27.755119 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e0499ec-8fcf-4308-918f-f5ac8086860c" path="/var/lib/kubelet/pods/3e0499ec-8fcf-4308-918f-f5ac8086860c/volumes" Oct 31 19:30:27.755622 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:27.755601 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5939df20-88c8-41de-bc5d-2bcd988afe95" path="/var/lib/kubelet/pods/5939df20-88c8-41de-bc5d-2bcd988afe95/volumes" Oct 31 19:30:27.756105 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:27.756080 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87ea32e0-73b4-4d40-857e-e2940bff400c" path="/var/lib/kubelet/pods/87ea32e0-73b4-4d40-857e-e2940bff400c/volumes" Oct 31 19:30:29.493330 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:29.493300 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:30:29.507239 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:29.507210 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:30:30.468378 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:30.468343 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:30:40.784781 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.784701 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e248b878-72fgf"] Oct 31 19:30:40.785314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785182 4910 cpu_manager.go:414] "containerMap: removing container" podUID="87ea32e0-73b4-4d40-857e-e2940bff400c" containerName="cleanup" Oct 31 19:30:40.785314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785195 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ea32e0-73b4-4d40-857e-e2940bff400c" containerName="cleanup" Oct 31 19:30:40.785314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785211 4910 cpu_manager.go:414] "containerMap: removing container" podUID="3e0499ec-8fcf-4308-918f-f5ac8086860c" containerName="cleanup" Oct 31 19:30:40.785314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785216 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e0499ec-8fcf-4308-918f-f5ac8086860c" containerName="cleanup" Oct 31 19:30:40.785314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785225 4910 cpu_manager.go:414] "containerMap: removing container" podUID="87ea32e0-73b4-4d40-857e-e2940bff400c" containerName="cleanup" Oct 31 19:30:40.785314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785231 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ea32e0-73b4-4d40-857e-e2940bff400c" containerName="cleanup" Oct 31 19:30:40.785314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785239 4910 cpu_manager.go:414] "containerMap: removing container" podUID="5939df20-88c8-41de-bc5d-2bcd988afe95" containerName="cleanup" Oct 31 19:30:40.785314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785244 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5939df20-88c8-41de-bc5d-2bcd988afe95" containerName="cleanup" Oct 31 19:30:40.785314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785250 4910 cpu_manager.go:414] "containerMap: removing container" podUID="84fc6fdb-08c7-49bf-9fa0-84ead1928480" containerName="cleanup" Oct 31 19:30:40.785314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785254 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84fc6fdb-08c7-49bf-9fa0-84ead1928480" containerName="cleanup" Oct 31 19:30:40.785314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785264 4910 cpu_manager.go:414] "containerMap: removing container" podUID="3e0499ec-8fcf-4308-918f-f5ac8086860c" containerName="cleanup" Oct 31 19:30:40.785314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785268 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e0499ec-8fcf-4308-918f-f5ac8086860c" containerName="cleanup" Oct 31 19:30:40.785314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785277 4910 cpu_manager.go:414] "containerMap: removing container" podUID="84fc6fdb-08c7-49bf-9fa0-84ead1928480" containerName="cleanup" Oct 31 19:30:40.785314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785283 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84fc6fdb-08c7-49bf-9fa0-84ead1928480" containerName="cleanup" Oct 31 19:30:40.785314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785294 4910 cpu_manager.go:414] "containerMap: removing container" podUID="87ea32e0-73b4-4d40-857e-e2940bff400c" containerName="cleanup" Oct 31 19:30:40.785314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785299 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87ea32e0-73b4-4d40-857e-e2940bff400c" containerName="cleanup" Oct 31 19:30:40.785314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785317 4910 cpu_manager.go:414] "containerMap: removing container" podUID="5939df20-88c8-41de-bc5d-2bcd988afe95" containerName="cleanup" Oct 31 19:30:40.785314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785322 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5939df20-88c8-41de-bc5d-2bcd988afe95" containerName="cleanup" Oct 31 19:30:40.785966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785402 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ea32e0-73b4-4d40-857e-e2940bff400c" containerName="cleanup" Oct 31 19:30:40.785966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785410 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e0499ec-8fcf-4308-918f-f5ac8086860c" containerName="cleanup" Oct 31 19:30:40.785966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785416 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="5939df20-88c8-41de-bc5d-2bcd988afe95" containerName="cleanup" Oct 31 19:30:40.785966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785421 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="84fc6fdb-08c7-49bf-9fa0-84ead1928480" containerName="cleanup" Oct 31 19:30:40.785966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785427 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e0499ec-8fcf-4308-918f-f5ac8086860c" containerName="cleanup" Oct 31 19:30:40.785966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785432 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="84fc6fdb-08c7-49bf-9fa0-84ead1928480" containerName="cleanup" Oct 31 19:30:40.785966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785437 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ea32e0-73b4-4d40-857e-e2940bff400c" containerName="cleanup" Oct 31 19:30:40.785966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.785446 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="5939df20-88c8-41de-bc5d-2bcd988afe95" containerName="cleanup" Oct 31 19:30:40.791732 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.791712 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e248b878-72fgf" Oct 31 19:30:40.796296 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.796249 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:30:40.797120 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.797099 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:30:40.797838 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.797815 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e248b878-72fgf"] Oct 31 19:30:40.865270 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.865231 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpcf9\" (UniqueName: \"kubernetes.io/projected/d941864a-96b8-4460-8fff-23abdc637c19-kube-api-access-bpcf9\") pod \"maas-api-revoke-sub-e248b878-72fgf\" (UID: \"d941864a-96b8-4460-8fff-23abdc637c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e248b878-72fgf" Oct 31 19:30:40.865450 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.865285 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d941864a-96b8-4460-8fff-23abdc637c19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e248b878-72fgf\" (UID: \"d941864a-96b8-4460-8fff-23abdc637c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e248b878-72fgf" Oct 31 19:30:40.865450 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.865366 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d941864a-96b8-4460-8fff-23abdc637c19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e248b878-72fgf\" (UID: \"d941864a-96b8-4460-8fff-23abdc637c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e248b878-72fgf" Oct 31 19:30:40.966221 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.966185 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpcf9\" (UniqueName: \"kubernetes.io/projected/d941864a-96b8-4460-8fff-23abdc637c19-kube-api-access-bpcf9\") pod \"maas-api-revoke-sub-e248b878-72fgf\" (UID: \"d941864a-96b8-4460-8fff-23abdc637c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e248b878-72fgf" Oct 31 19:30:40.966422 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.966245 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d941864a-96b8-4460-8fff-23abdc637c19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e248b878-72fgf\" (UID: \"d941864a-96b8-4460-8fff-23abdc637c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e248b878-72fgf" Oct 31 19:30:40.966422 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.966293 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d941864a-96b8-4460-8fff-23abdc637c19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e248b878-72fgf\" (UID: \"d941864a-96b8-4460-8fff-23abdc637c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e248b878-72fgf" Oct 31 19:30:40.967041 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.967015 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d941864a-96b8-4460-8fff-23abdc637c19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e248b878-72fgf\" (UID: \"d941864a-96b8-4460-8fff-23abdc637c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e248b878-72fgf" Oct 31 19:30:40.968552 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.968529 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d941864a-96b8-4460-8fff-23abdc637c19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e248b878-72fgf\" (UID: \"d941864a-96b8-4460-8fff-23abdc637c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e248b878-72fgf" Oct 31 19:30:40.979339 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:40.979313 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpcf9\" (UniqueName: \"kubernetes.io/projected/d941864a-96b8-4460-8fff-23abdc637c19-kube-api-access-bpcf9\") pod \"maas-api-revoke-sub-e248b878-72fgf\" (UID: \"d941864a-96b8-4460-8fff-23abdc637c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e248b878-72fgf" Oct 31 19:30:41.110068 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:41.110031 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e248b878-72fgf" Oct 31 19:30:41.398187 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:41.398162 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e248b878-72fgf"] Oct 31 19:30:41.398619 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:30:41.398590 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd941864a_96b8_4460_8fff_23abdc637c19.slice/crio-7a07b39796388b1d0e8f3b96c467816c53d63777f770fb7fb84adfe5b6350f12 WatchSource:0}: Error finding container 7a07b39796388b1d0e8f3b96c467816c53d63777f770fb7fb84adfe5b6350f12: Status 404 returned error can't find the container with id 7a07b39796388b1d0e8f3b96c467816c53d63777f770fb7fb84adfe5b6350f12 Oct 31 19:30:41.535020 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:41.534982 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e248b878-72fgf" event={"ID":"d941864a-96b8-4460-8fff-23abdc637c19","Type":"ContainerStarted","Data":"7a07b39796388b1d0e8f3b96c467816c53d63777f770fb7fb84adfe5b6350f12"} Oct 31 19:30:42.545165 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:42.545128 4910 generic.go:356] "Generic (PLEG): container finished" podID="d941864a-96b8-4460-8fff-23abdc637c19" containerID="c65c970bd8053e84f6c5dafc387a56afea05c274667d6a00f48183e60c228558" exitCode=0 Oct 31 19:30:42.545580 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:42.545221 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e248b878-72fgf" event={"ID":"d941864a-96b8-4460-8fff-23abdc637c19","Type":"ContainerDied","Data":"c65c970bd8053e84f6c5dafc387a56afea05c274667d6a00f48183e60c228558"} Oct 31 19:30:42.711866 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:30:42.711781 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:30:43.710649 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:43.710623 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e248b878-72fgf" Oct 31 19:30:43.798667 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:43.798637 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d941864a-96b8-4460-8fff-23abdc637c19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d941864a-96b8-4460-8fff-23abdc637c19-maas-api-xks-ca\") pod \"d941864a-96b8-4460-8fff-23abdc637c19\" (UID: \"d941864a-96b8-4460-8fff-23abdc637c19\") " Oct 31 19:30:43.798844 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:43.798761 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d941864a-96b8-4460-8fff-23abdc637c19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d941864a-96b8-4460-8fff-23abdc637c19-maas-api-openshift-ca\") pod \"d941864a-96b8-4460-8fff-23abdc637c19\" (UID: \"d941864a-96b8-4460-8fff-23abdc637c19\") " Oct 31 19:30:43.798992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:43.798959 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d941864a-96b8-4460-8fff-23abdc637c19-kube-api-access-bpcf9\" (UniqueName: \"kubernetes.io/projected/d941864a-96b8-4460-8fff-23abdc637c19-kube-api-access-bpcf9\") pod \"d941864a-96b8-4460-8fff-23abdc637c19\" (UID: \"d941864a-96b8-4460-8fff-23abdc637c19\") " Oct 31 19:30:43.799978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:43.799897 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d941864a-96b8-4460-8fff-23abdc637c19-maas-api-openshift-ca" pod "d941864a-96b8-4460-8fff-23abdc637c19" (UID: "d941864a-96b8-4460-8fff-23abdc637c19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:30:43.801508 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:43.801483 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d941864a-96b8-4460-8fff-23abdc637c19-maas-api-xks-ca" pod "d941864a-96b8-4460-8fff-23abdc637c19" (UID: "d941864a-96b8-4460-8fff-23abdc637c19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:30:43.801723 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:43.801698 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d941864a-96b8-4460-8fff-23abdc637c19-kube-api-access-bpcf9" pod "d941864a-96b8-4460-8fff-23abdc637c19" (UID: "d941864a-96b8-4460-8fff-23abdc637c19"). InnerVolumeSpecName "kube-api-access-bpcf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:30:43.900375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:43.900346 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpcf9\" (UniqueName: \"kubernetes.io/projected/d941864a-96b8-4460-8fff-23abdc637c19-kube-api-access-bpcf9\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:30:43.900375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:43.900372 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d941864a-96b8-4460-8fff-23abdc637c19-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:30:43.900375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:43.900383 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d941864a-96b8-4460-8fff-23abdc637c19-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:30:44.572755 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:44.572661 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e248b878-72fgf" event={"ID":"d941864a-96b8-4460-8fff-23abdc637c19","Type":"ContainerDied","Data":"7a07b39796388b1d0e8f3b96c467816c53d63777f770fb7fb84adfe5b6350f12"} Oct 31 19:30:44.572755 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:44.572731 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a07b39796388b1d0e8f3b96c467816c53d63777f770fb7fb84adfe5b6350f12" Oct 31 19:30:44.573133 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:44.572826 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e248b878-72fgf" Oct 31 19:30:47.909980 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:47.909932 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48"] Oct 31 19:30:47.910404 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:47.910389 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d941864a-96b8-4460-8fff-23abdc637c19" containerName="revoke-keys" Oct 31 19:30:47.910447 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:47.910405 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d941864a-96b8-4460-8fff-23abdc637c19" containerName="revoke-keys" Oct 31 19:30:47.910447 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:47.910418 4910 cpu_manager.go:414] "containerMap: removing container" podUID="5939df20-88c8-41de-bc5d-2bcd988afe95" containerName="cleanup" Oct 31 19:30:47.910447 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:47.910438 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5939df20-88c8-41de-bc5d-2bcd988afe95" containerName="cleanup" Oct 31 19:30:47.910535 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:47.910450 4910 cpu_manager.go:414] "containerMap: removing container" podUID="3e0499ec-8fcf-4308-918f-f5ac8086860c" containerName="cleanup" Oct 31 19:30:47.910535 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:47.910455 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e0499ec-8fcf-4308-918f-f5ac8086860c" containerName="cleanup" Oct 31 19:30:47.910535 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:47.910465 4910 cpu_manager.go:414] "containerMap: removing container" podUID="84fc6fdb-08c7-49bf-9fa0-84ead1928480" containerName="cleanup" Oct 31 19:30:47.910535 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:47.910470 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84fc6fdb-08c7-49bf-9fa0-84ead1928480" containerName="cleanup" Oct 31 19:30:47.910651 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:47.910548 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d941864a-96b8-4460-8fff-23abdc637c19" containerName="revoke-keys" Oct 31 19:30:47.910651 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:47.910559 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="5939df20-88c8-41de-bc5d-2bcd988afe95" containerName="cleanup" Oct 31 19:30:47.910651 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:47.910568 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="84fc6fdb-08c7-49bf-9fa0-84ead1928480" containerName="cleanup" Oct 31 19:30:47.910651 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:47.910574 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e0499ec-8fcf-4308-918f-f5ac8086860c" containerName="cleanup" Oct 31 19:30:47.910651 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:47.910580 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="87ea32e0-73b4-4d40-857e-e2940bff400c" containerName="cleanup" Oct 31 19:30:47.914255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:47.914232 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:47.919514 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:47.919481 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-aeb292-openshift-default-dockercfg-zq554\"" Oct 31 19:30:47.928022 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:47.927995 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48"] Oct 31 19:30:48.048803 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.048767 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.049022 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.048816 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.049022 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.048895 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.049022 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.048972 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.049022 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.048995 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.049257 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.049094 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.049257 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.049113 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.049257 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.049131 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz6ts\" (UniqueName: \"kubernetes.io/projected/cc8a6def-fe67-42b8-aea6-b3cd2399def1-kube-api-access-dz6ts\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.049257 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.049157 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.150758 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.150722 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.150978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.150780 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.150978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.150810 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.150978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.150849 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.150978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.150883 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.151221 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.150983 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.151221 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.151009 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.151221 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.151035 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz6ts\" (UniqueName: \"kubernetes.io/projected/cc8a6def-fe67-42b8-aea6-b3cd2399def1-kube-api-access-dz6ts\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.151221 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.151074 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.151397 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.151235 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.151450 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.151412 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.151691 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.151667 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.151950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.151880 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.152118 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.152077 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.153723 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.153701 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.153803 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.153758 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.163808 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.163737 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.164339 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.164312 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz6ts\" (UniqueName: \"kubernetes.io/projected/cc8a6def-fe67-42b8-aea6-b3cd2399def1-kube-api-access-dz6ts\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.230595 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.230561 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:48.518078 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.518046 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48"] Oct 31 19:30:48.518554 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:30:48.518531 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8a6def_fe67_42b8_aea6_b3cd2399def1.slice/crio-f29a8bd6c9a80ccaae949596109d7148d9316d0bdcd8e9b796005cd9ca319de4 WatchSource:0}: Error finding container f29a8bd6c9a80ccaae949596109d7148d9316d0bdcd8e9b796005cd9ca319de4: Status 404 returned error can't find the container with id f29a8bd6c9a80ccaae949596109d7148d9316d0bdcd8e9b796005cd9ca319de4 Oct 31 19:30:48.521298 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.521261 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:30:48.521401 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.521340 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:30:48.521401 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.521375 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:30:48.560391 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.560364 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t"] Oct 31 19:30:48.565499 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.565476 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.576471 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.576454 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-23814d-openshift-default-dockercfg-m6pw6\"" Oct 31 19:30:48.581783 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.581757 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t"] Oct 31 19:30:48.602642 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.602616 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" event={"ID":"cc8a6def-fe67-42b8-aea6-b3cd2399def1","Type":"ContainerStarted","Data":"f29a8bd6c9a80ccaae949596109d7148d9316d0bdcd8e9b796005cd9ca319de4"} Oct 31 19:30:48.759049 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.759014 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.759049 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.759050 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.759310 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.759189 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.759310 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.759263 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.759310 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.759284 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.759468 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.759328 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d970e7-85b8-4057-8b20-5df84679b3ab-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.759468 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.759354 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.759468 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.759403 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf6vw\" (UniqueName: \"kubernetes.io/projected/25d970e7-85b8-4057-8b20-5df84679b3ab-kube-api-access-kf6vw\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.759625 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.759485 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.860490 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.860459 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.860490 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.860490 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.860743 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.860524 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.860743 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.860580 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.860743 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.860608 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.860743 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.860635 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d970e7-85b8-4057-8b20-5df84679b3ab-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.860743 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.860675 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.860743 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.860698 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf6vw\" (UniqueName: \"kubernetes.io/projected/25d970e7-85b8-4057-8b20-5df84679b3ab-kube-api-access-kf6vw\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.861074 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.860809 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.861074 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.860887 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.861280 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.861238 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.861358 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.861336 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.861526 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.861504 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d970e7-85b8-4057-8b20-5df84679b3ab-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.861868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.861851 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.863788 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.863766 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.864197 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.864175 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.875350 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.875329 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.875458 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.875374 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf6vw\" (UniqueName: \"kubernetes.io/projected/25d970e7-85b8-4057-8b20-5df84679b3ab-kube-api-access-kf6vw\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:48.901080 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:48.901057 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:49.199159 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:49.199074 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t"] Oct 31 19:30:49.211217 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:49.211173 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:30:49.211333 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:49.211266 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:30:49.211333 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:49.211308 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:30:49.459495 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:49.458985 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d119c4a7-dxl8c"] Oct 31 19:30:49.464272 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:49.464238 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d119c4a7-dxl8c"] Oct 31 19:30:49.464595 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:49.464574 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d119c4a7-dxl8c" Oct 31 19:30:49.475489 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:49.473619 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:30:49.475489 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:49.473695 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:30:49.573821 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:49.573778 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4wh4\" (UniqueName: \"kubernetes.io/projected/b07f8f0e-9ef5-4609-9302-39562f5f9074-kube-api-access-t4wh4\") pod \"maas-api-revoke-sub-d119c4a7-dxl8c\" (UID: \"b07f8f0e-9ef5-4609-9302-39562f5f9074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d119c4a7-dxl8c" Oct 31 19:30:49.574046 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:49.573942 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07f8f0e-9ef5-4609-9302-39562f5f9074-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d119c4a7-dxl8c\" (UID: \"b07f8f0e-9ef5-4609-9302-39562f5f9074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d119c4a7-dxl8c" Oct 31 19:30:49.574178 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:49.574159 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07f8f0e-9ef5-4609-9302-39562f5f9074-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d119c4a7-dxl8c\" (UID: \"b07f8f0e-9ef5-4609-9302-39562f5f9074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d119c4a7-dxl8c" Oct 31 19:30:49.611264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:49.611228 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" event={"ID":"cc8a6def-fe67-42b8-aea6-b3cd2399def1","Type":"ContainerStarted","Data":"f3b13311a7417606d2db084c1931e25fa6191b2867223543b624a4494dd0138d"} Oct 31 19:30:49.613008 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:49.612980 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" event={"ID":"25d970e7-85b8-4057-8b20-5df84679b3ab","Type":"ContainerStarted","Data":"81d1f5dc97000d0eeb0a8510c17f6c12563b497d0dafe57db65ddacc09daa26d"} Oct 31 19:30:49.613128 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:49.613031 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" event={"ID":"25d970e7-85b8-4057-8b20-5df84679b3ab","Type":"ContainerStarted","Data":"41fab0b24e083d02b5ac83458e968ee9bffa9f6ff3ee31504d457f098f9b67d9"} Oct 31 19:30:49.666758 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:49.666670 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" podStartSLOduration=2.666652039 podStartE2EDuration="2.666652039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:49.66015692 +0000 UTC m=+684.022977314" watchObservedRunningTime="2026-10-31 19:30:49.666652039 +0000 UTC m=+684.029472411" Oct 31 19:30:49.674827 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:49.674793 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07f8f0e-9ef5-4609-9302-39562f5f9074-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d119c4a7-dxl8c\" (UID: \"b07f8f0e-9ef5-4609-9302-39562f5f9074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d119c4a7-dxl8c" Oct 31 19:30:49.675045 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:49.674853 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4wh4\" (UniqueName: \"kubernetes.io/projected/b07f8f0e-9ef5-4609-9302-39562f5f9074-kube-api-access-t4wh4\") pod \"maas-api-revoke-sub-d119c4a7-dxl8c\" (UID: \"b07f8f0e-9ef5-4609-9302-39562f5f9074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d119c4a7-dxl8c" Oct 31 19:30:49.675045 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:49.674892 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07f8f0e-9ef5-4609-9302-39562f5f9074-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d119c4a7-dxl8c\" (UID: \"b07f8f0e-9ef5-4609-9302-39562f5f9074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d119c4a7-dxl8c" Oct 31 19:30:49.675495 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:49.675466 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07f8f0e-9ef5-4609-9302-39562f5f9074-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d119c4a7-dxl8c\" (UID: \"b07f8f0e-9ef5-4609-9302-39562f5f9074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d119c4a7-dxl8c" Oct 31 19:30:49.677428 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:49.677404 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07f8f0e-9ef5-4609-9302-39562f5f9074-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d119c4a7-dxl8c\" (UID: \"b07f8f0e-9ef5-4609-9302-39562f5f9074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d119c4a7-dxl8c" Oct 31 19:30:49.687086 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:49.687032 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" podStartSLOduration=1.687019967 podStartE2EDuration="1.687019967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:49.683971836 +0000 UTC m=+684.046792209" watchObservedRunningTime="2026-10-31 19:30:49.687019967 +0000 UTC m=+684.049840340" Oct 31 19:30:49.691711 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:49.691687 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4wh4\" (UniqueName: \"kubernetes.io/projected/b07f8f0e-9ef5-4609-9302-39562f5f9074-kube-api-access-t4wh4\") pod \"maas-api-revoke-sub-d119c4a7-dxl8c\" (UID: \"b07f8f0e-9ef5-4609-9302-39562f5f9074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d119c4a7-dxl8c" Oct 31 19:30:49.791446 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:49.791363 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d119c4a7-dxl8c" Oct 31 19:30:49.901403 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:49.901370 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:50.108226 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:50.107874 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d119c4a7-dxl8c"] Oct 31 19:30:50.110293 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:30:50.110252 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07f8f0e_9ef5_4609_9302_39562f5f9074.slice/crio-afae0494c9c26154e27d70fedf20da4edcff4df80bdcf125d135bf85c139f27f WatchSource:0}: Error finding container afae0494c9c26154e27d70fedf20da4edcff4df80bdcf125d135bf85c139f27f: Status 404 returned error can't find the container with id afae0494c9c26154e27d70fedf20da4edcff4df80bdcf125d135bf85c139f27f Oct 31 19:30:50.233329 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:50.233254 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:50.257460 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:50.257079 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 31 19:30:50.257460 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:50.257144 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" podUID="cc8a6def-fe67-42b8-aea6-b3cd2399def1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 31 19:30:50.636942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:50.635510 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d119c4a7-dxl8c" event={"ID":"b07f8f0e-9ef5-4609-9302-39562f5f9074","Type":"ContainerStarted","Data":"afae0494c9c26154e27d70fedf20da4edcff4df80bdcf125d135bf85c139f27f"} Oct 31 19:30:50.903584 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:50.903527 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 31 19:30:50.903764 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:50.903628 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" podUID="25d970e7-85b8-4057-8b20-5df84679b3ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 31 19:30:51.237987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:51.236327 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 31 19:30:51.237987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:51.236390 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" podUID="cc8a6def-fe67-42b8-aea6-b3cd2399def1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 31 19:30:51.672665 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:51.672628 4910 generic.go:356] "Generic (PLEG): container finished" podID="b07f8f0e-9ef5-4609-9302-39562f5f9074" containerID="acb48e1267e383e7679757fd7d0c3777809c5a9cb91ceaf39560016edd44dd42" exitCode=0 Oct 31 19:30:51.672962 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:51.672722 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d119c4a7-dxl8c" event={"ID":"b07f8f0e-9ef5-4609-9302-39562f5f9074","Type":"ContainerDied","Data":"acb48e1267e383e7679757fd7d0c3777809c5a9cb91ceaf39560016edd44dd42"} Oct 31 19:30:51.902200 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:51.902157 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 31 19:30:51.902370 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:51.902224 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" podUID="25d970e7-85b8-4057-8b20-5df84679b3ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 31 19:30:52.235004 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:52.234966 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 31 19:30:52.235295 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:52.235262 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" podUID="cc8a6def-fe67-42b8-aea6-b3cd2399def1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 31 19:30:52.902939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:52.902424 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 31 19:30:52.902939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:52.902481 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" podUID="25d970e7-85b8-4057-8b20-5df84679b3ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 31 19:30:53.012180 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.012152 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d119c4a7-dxl8c" Oct 31 19:30:53.074933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.074349 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b07f8f0e-9ef5-4609-9302-39562f5f9074-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07f8f0e-9ef5-4609-9302-39562f5f9074-maas-api-xks-ca\") pod \"b07f8f0e-9ef5-4609-9302-39562f5f9074\" (UID: \"b07f8f0e-9ef5-4609-9302-39562f5f9074\") " Oct 31 19:30:53.074933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.074441 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b07f8f0e-9ef5-4609-9302-39562f5f9074-kube-api-access-t4wh4\" (UniqueName: \"kubernetes.io/projected/b07f8f0e-9ef5-4609-9302-39562f5f9074-kube-api-access-t4wh4\") pod \"b07f8f0e-9ef5-4609-9302-39562f5f9074\" (UID: \"b07f8f0e-9ef5-4609-9302-39562f5f9074\") " Oct 31 19:30:53.074933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.074517 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b07f8f0e-9ef5-4609-9302-39562f5f9074-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07f8f0e-9ef5-4609-9302-39562f5f9074-maas-api-openshift-ca\") pod \"b07f8f0e-9ef5-4609-9302-39562f5f9074\" (UID: \"b07f8f0e-9ef5-4609-9302-39562f5f9074\") " Oct 31 19:30:53.077961 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.076864 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b07f8f0e-9ef5-4609-9302-39562f5f9074-maas-api-openshift-ca" pod "b07f8f0e-9ef5-4609-9302-39562f5f9074" (UID: "b07f8f0e-9ef5-4609-9302-39562f5f9074"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:30:53.083896 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.083282 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b07f8f0e-9ef5-4609-9302-39562f5f9074-maas-api-xks-ca" pod "b07f8f0e-9ef5-4609-9302-39562f5f9074" (UID: "b07f8f0e-9ef5-4609-9302-39562f5f9074"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:30:53.083896 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.083831 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b07f8f0e-9ef5-4609-9302-39562f5f9074-kube-api-access-t4wh4" pod "b07f8f0e-9ef5-4609-9302-39562f5f9074" (UID: "b07f8f0e-9ef5-4609-9302-39562f5f9074"). InnerVolumeSpecName "kube-api-access-t4wh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:30:53.178267 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.178221 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07f8f0e-9ef5-4609-9302-39562f5f9074-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:30:53.178267 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.178270 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07f8f0e-9ef5-4609-9302-39562f5f9074-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:30:53.178469 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.178285 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4wh4\" (UniqueName: \"kubernetes.io/projected/b07f8f0e-9ef5-4609-9302-39562f5f9074-kube-api-access-t4wh4\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:30:53.238989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.238285 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 31 19:30:53.238989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.238355 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" podUID="cc8a6def-fe67-42b8-aea6-b3cd2399def1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 31 19:30:53.278934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.278010 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60a52118-k8fzb"] Oct 31 19:30:53.278934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.278749 4910 cpu_manager.go:414] "containerMap: removing container" podUID="b07f8f0e-9ef5-4609-9302-39562f5f9074" containerName="revoke-keys" Oct 31 19:30:53.278934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.278773 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b07f8f0e-9ef5-4609-9302-39562f5f9074" containerName="revoke-keys" Oct 31 19:30:53.279223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.278970 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="b07f8f0e-9ef5-4609-9302-39562f5f9074" containerName="revoke-keys" Oct 31 19:30:53.284662 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.283756 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a52118-k8fzb" Oct 31 19:30:53.292001 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.291761 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60a52118-k8fzb"] Oct 31 19:30:53.381551 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.381520 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044955bb-e4be-49d1-9dfd-5500fb1094d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60a52118-k8fzb\" (UID: \"044955bb-e4be-49d1-9dfd-5500fb1094d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a52118-k8fzb" Oct 31 19:30:53.381718 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.381564 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xd7x\" (UniqueName: \"kubernetes.io/projected/044955bb-e4be-49d1-9dfd-5500fb1094d5-kube-api-access-4xd7x\") pod \"maas-api-revoke-sub-60a52118-k8fzb\" (UID: \"044955bb-e4be-49d1-9dfd-5500fb1094d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a52118-k8fzb" Oct 31 19:30:53.381774 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.381760 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044955bb-e4be-49d1-9dfd-5500fb1094d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60a52118-k8fzb\" (UID: \"044955bb-e4be-49d1-9dfd-5500fb1094d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a52118-k8fzb" Oct 31 19:30:53.482738 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.482702 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044955bb-e4be-49d1-9dfd-5500fb1094d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60a52118-k8fzb\" (UID: \"044955bb-e4be-49d1-9dfd-5500fb1094d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a52118-k8fzb" Oct 31 19:30:53.482973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.482795 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044955bb-e4be-49d1-9dfd-5500fb1094d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60a52118-k8fzb\" (UID: \"044955bb-e4be-49d1-9dfd-5500fb1094d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a52118-k8fzb" Oct 31 19:30:53.482973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.482823 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xd7x\" (UniqueName: \"kubernetes.io/projected/044955bb-e4be-49d1-9dfd-5500fb1094d5-kube-api-access-4xd7x\") pod \"maas-api-revoke-sub-60a52118-k8fzb\" (UID: \"044955bb-e4be-49d1-9dfd-5500fb1094d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a52118-k8fzb" Oct 31 19:30:53.483610 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.483584 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044955bb-e4be-49d1-9dfd-5500fb1094d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60a52118-k8fzb\" (UID: \"044955bb-e4be-49d1-9dfd-5500fb1094d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a52118-k8fzb" Oct 31 19:30:53.485480 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.485444 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044955bb-e4be-49d1-9dfd-5500fb1094d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60a52118-k8fzb\" (UID: \"044955bb-e4be-49d1-9dfd-5500fb1094d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a52118-k8fzb" Oct 31 19:30:53.497231 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.497173 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xd7x\" (UniqueName: \"kubernetes.io/projected/044955bb-e4be-49d1-9dfd-5500fb1094d5-kube-api-access-4xd7x\") pod \"maas-api-revoke-sub-60a52118-k8fzb\" (UID: \"044955bb-e4be-49d1-9dfd-5500fb1094d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a52118-k8fzb" Oct 31 19:30:53.602955 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.602904 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a52118-k8fzb" Oct 31 19:30:53.687633 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.687603 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d119c4a7-dxl8c" Oct 31 19:30:53.687861 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.687634 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d119c4a7-dxl8c" event={"ID":"b07f8f0e-9ef5-4609-9302-39562f5f9074","Type":"ContainerDied","Data":"afae0494c9c26154e27d70fedf20da4edcff4df80bdcf125d135bf85c139f27f"} Oct 31 19:30:53.687983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.687964 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afae0494c9c26154e27d70fedf20da4edcff4df80bdcf125d135bf85c139f27f" Oct 31 19:30:53.901893 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.901859 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 31 19:30:53.902204 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.902171 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" podUID="25d970e7-85b8-4057-8b20-5df84679b3ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 31 19:30:53.980455 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:53.980406 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60a52118-k8fzb"] Oct 31 19:30:53.987052 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:30:53.986589 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044955bb_e4be_49d1_9dfd_5500fb1094d5.slice/crio-fdc4d60e3d2146a4868fe356302b4f09473169c13498c761843380e6d59496cd WatchSource:0}: Error finding container fdc4d60e3d2146a4868fe356302b4f09473169c13498c761843380e6d59496cd: Status 404 returned error can't find the container with id fdc4d60e3d2146a4868fe356302b4f09473169c13498c761843380e6d59496cd Oct 31 19:30:54.250993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:54.250351 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 31 19:30:54.250993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:54.250445 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" podUID="cc8a6def-fe67-42b8-aea6-b3cd2399def1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 31 19:30:54.696839 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:54.696803 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a52118-k8fzb" event={"ID":"044955bb-e4be-49d1-9dfd-5500fb1094d5","Type":"ContainerStarted","Data":"3e2bea4e07f6b0b6a43f4e3dc0182d3d5723eb214ddd004fe3d5f8d2eafe0900"} Oct 31 19:30:54.697015 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:54.696869 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a52118-k8fzb" event={"ID":"044955bb-e4be-49d1-9dfd-5500fb1094d5","Type":"ContainerStarted","Data":"fdc4d60e3d2146a4868fe356302b4f09473169c13498c761843380e6d59496cd"} Oct 31 19:30:54.716127 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:54.715986 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a52118-k8fzb" podStartSLOduration=1.715967414 podStartE2EDuration="1.715967414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:54.715025159 +0000 UTC m=+689.077845531" watchObservedRunningTime="2026-10-31 19:30:54.715967414 +0000 UTC m=+689.078787787" Oct 31 19:30:54.902213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:54.902177 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 31 19:30:54.902381 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:54.902244 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" podUID="25d970e7-85b8-4057-8b20-5df84679b3ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 31 19:30:55.231938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:55.231325 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 31 19:30:55.231938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:55.231391 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" podUID="cc8a6def-fe67-42b8-aea6-b3cd2399def1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 31 19:30:55.705537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:55.705494 4910 generic.go:356] "Generic (PLEG): container finished" podID="044955bb-e4be-49d1-9dfd-5500fb1094d5" containerID="3e2bea4e07f6b0b6a43f4e3dc0182d3d5723eb214ddd004fe3d5f8d2eafe0900" exitCode=0 Oct 31 19:30:55.705726 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:55.705578 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a52118-k8fzb" event={"ID":"044955bb-e4be-49d1-9dfd-5500fb1094d5","Type":"ContainerDied","Data":"3e2bea4e07f6b0b6a43f4e3dc0182d3d5723eb214ddd004fe3d5f8d2eafe0900"} Oct 31 19:30:55.901684 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:55.901652 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 31 19:30:55.901868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:55.901707 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" podUID="25d970e7-85b8-4057-8b20-5df84679b3ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 31 19:30:56.236951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:56.236893 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:56.237441 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:56.237283 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:56.238177 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:56.238149 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:30:56.876035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:56.876007 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a52118-k8fzb" Oct 31 19:30:56.907056 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:56.907022 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:56.910938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:56.908178 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:56.920272 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:56.920245 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:30:56.937115 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:56.937081 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/044955bb-e4be-49d1-9dfd-5500fb1094d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044955bb-e4be-49d1-9dfd-5500fb1094d5-maas-api-xks-ca\") pod \"044955bb-e4be-49d1-9dfd-5500fb1094d5\" (UID: \"044955bb-e4be-49d1-9dfd-5500fb1094d5\") " Oct 31 19:30:56.937280 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:56.937120 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/044955bb-e4be-49d1-9dfd-5500fb1094d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044955bb-e4be-49d1-9dfd-5500fb1094d5-maas-api-openshift-ca\") pod \"044955bb-e4be-49d1-9dfd-5500fb1094d5\" (UID: \"044955bb-e4be-49d1-9dfd-5500fb1094d5\") " Oct 31 19:30:56.937280 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:56.937197 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/044955bb-e4be-49d1-9dfd-5500fb1094d5-kube-api-access-4xd7x\" (UniqueName: \"kubernetes.io/projected/044955bb-e4be-49d1-9dfd-5500fb1094d5-kube-api-access-4xd7x\") pod \"044955bb-e4be-49d1-9dfd-5500fb1094d5\" (UID: \"044955bb-e4be-49d1-9dfd-5500fb1094d5\") " Oct 31 19:30:56.937577 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:56.937551 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/044955bb-e4be-49d1-9dfd-5500fb1094d5-maas-api-openshift-ca" pod "044955bb-e4be-49d1-9dfd-5500fb1094d5" (UID: "044955bb-e4be-49d1-9dfd-5500fb1094d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:30:56.937832 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:56.937808 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044955bb-e4be-49d1-9dfd-5500fb1094d5-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:30:56.940090 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:56.940061 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/044955bb-e4be-49d1-9dfd-5500fb1094d5-maas-api-xks-ca" pod "044955bb-e4be-49d1-9dfd-5500fb1094d5" (UID: "044955bb-e4be-49d1-9dfd-5500fb1094d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:30:56.944389 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:56.944360 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044955bb-e4be-49d1-9dfd-5500fb1094d5-kube-api-access-4xd7x" pod "044955bb-e4be-49d1-9dfd-5500fb1094d5" (UID: "044955bb-e4be-49d1-9dfd-5500fb1094d5"). InnerVolumeSpecName "kube-api-access-4xd7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:30:57.039376 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:57.039291 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044955bb-e4be-49d1-9dfd-5500fb1094d5-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:30:57.039376 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:57.039321 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xd7x\" (UniqueName: \"kubernetes.io/projected/044955bb-e4be-49d1-9dfd-5500fb1094d5-kube-api-access-4xd7x\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:30:57.721245 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:57.721212 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a52118-k8fzb" Oct 31 19:30:57.721834 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:57.721240 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a52118-k8fzb" event={"ID":"044955bb-e4be-49d1-9dfd-5500fb1094d5","Type":"ContainerDied","Data":"fdc4d60e3d2146a4868fe356302b4f09473169c13498c761843380e6d59496cd"} Oct 31 19:30:57.721834 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:30:57.721300 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdc4d60e3d2146a4868fe356302b4f09473169c13498c761843380e6d59496cd" Oct 31 19:31:01.765980 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:01.765945 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fed47bff-kz7g9"] Oct 31 19:31:01.766410 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:01.766398 4910 cpu_manager.go:414] "containerMap: removing container" podUID="044955bb-e4be-49d1-9dfd-5500fb1094d5" containerName="revoke-keys" Oct 31 19:31:01.766453 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:01.766411 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="044955bb-e4be-49d1-9dfd-5500fb1094d5" containerName="revoke-keys" Oct 31 19:31:01.766509 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:01.766501 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="044955bb-e4be-49d1-9dfd-5500fb1094d5" containerName="revoke-keys" Oct 31 19:31:01.770259 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:01.770236 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed47bff-kz7g9" Oct 31 19:31:01.777021 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:01.776982 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:31:01.778891 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:01.778867 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:31:01.783598 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:01.783574 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fed47bff-kz7g9"] Oct 31 19:31:01.815470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:01.815443 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab24981d-834d-49d0-bfa7-eb64ff0ef2d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fed47bff-kz7g9\" (UID: \"ab24981d-834d-49d0-bfa7-eb64ff0ef2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed47bff-kz7g9" Oct 31 19:31:01.815615 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:01.815486 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab24981d-834d-49d0-bfa7-eb64ff0ef2d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fed47bff-kz7g9\" (UID: \"ab24981d-834d-49d0-bfa7-eb64ff0ef2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed47bff-kz7g9" Oct 31 19:31:01.815615 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:01.815520 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v59hv\" (UniqueName: \"kubernetes.io/projected/ab24981d-834d-49d0-bfa7-eb64ff0ef2d4-kube-api-access-v59hv\") pod \"maas-api-revoke-sub-fed47bff-kz7g9\" (UID: \"ab24981d-834d-49d0-bfa7-eb64ff0ef2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed47bff-kz7g9" Oct 31 19:31:01.916801 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:01.916770 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab24981d-834d-49d0-bfa7-eb64ff0ef2d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fed47bff-kz7g9\" (UID: \"ab24981d-834d-49d0-bfa7-eb64ff0ef2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed47bff-kz7g9" Oct 31 19:31:01.917023 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:01.916817 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab24981d-834d-49d0-bfa7-eb64ff0ef2d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fed47bff-kz7g9\" (UID: \"ab24981d-834d-49d0-bfa7-eb64ff0ef2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed47bff-kz7g9" Oct 31 19:31:01.917023 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:01.916847 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v59hv\" (UniqueName: \"kubernetes.io/projected/ab24981d-834d-49d0-bfa7-eb64ff0ef2d4-kube-api-access-v59hv\") pod \"maas-api-revoke-sub-fed47bff-kz7g9\" (UID: \"ab24981d-834d-49d0-bfa7-eb64ff0ef2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed47bff-kz7g9" Oct 31 19:31:01.917595 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:01.917566 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab24981d-834d-49d0-bfa7-eb64ff0ef2d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fed47bff-kz7g9\" (UID: \"ab24981d-834d-49d0-bfa7-eb64ff0ef2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed47bff-kz7g9" Oct 31 19:31:01.919217 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:01.919194 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab24981d-834d-49d0-bfa7-eb64ff0ef2d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fed47bff-kz7g9\" (UID: \"ab24981d-834d-49d0-bfa7-eb64ff0ef2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed47bff-kz7g9" Oct 31 19:31:01.930542 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:01.930518 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v59hv\" (UniqueName: \"kubernetes.io/projected/ab24981d-834d-49d0-bfa7-eb64ff0ef2d4-kube-api-access-v59hv\") pod \"maas-api-revoke-sub-fed47bff-kz7g9\" (UID: \"ab24981d-834d-49d0-bfa7-eb64ff0ef2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed47bff-kz7g9" Oct 31 19:31:02.086608 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:02.086575 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed47bff-kz7g9" Oct 31 19:31:02.369214 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:31:02.369186 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab24981d_834d_49d0_bfa7_eb64ff0ef2d4.slice/crio-1850c04adbad8812988a698f27c3deb235ce004318ee00cee75dd98c6ce14c19 WatchSource:0}: Error finding container 1850c04adbad8812988a698f27c3deb235ce004318ee00cee75dd98c6ce14c19: Status 404 returned error can't find the container with id 1850c04adbad8812988a698f27c3deb235ce004318ee00cee75dd98c6ce14c19 Oct 31 19:31:02.369615 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:02.369593 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fed47bff-kz7g9"] Oct 31 19:31:02.790875 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:02.790835 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed47bff-kz7g9" event={"ID":"ab24981d-834d-49d0-bfa7-eb64ff0ef2d4","Type":"ContainerStarted","Data":"4f759eb2ccece62f8ce8084cac067123e3a2a63fe10fa840a5695ec89b3190da"} Oct 31 19:31:02.791395 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:02.790899 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed47bff-kz7g9" event={"ID":"ab24981d-834d-49d0-bfa7-eb64ff0ef2d4","Type":"ContainerStarted","Data":"1850c04adbad8812988a698f27c3deb235ce004318ee00cee75dd98c6ce14c19"} Oct 31 19:31:02.825575 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:02.825525 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed47bff-kz7g9" podStartSLOduration=1.825511015 podStartE2EDuration="1.825511015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:02.822282713 +0000 UTC m=+697.185103087" watchObservedRunningTime="2026-10-31 19:31:02.825511015 +0000 UTC m=+697.188331423" Oct 31 19:31:03.391796 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.391717 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z"] Oct 31 19:31:03.395524 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.395501 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.399707 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.399677 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-8730d5-openshift-default-dockercfg-hc6b7\"" Oct 31 19:31:03.408767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.408744 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z"] Oct 31 19:31:03.433142 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.433102 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.433142 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.433145 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.433354 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.433259 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.433354 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.433309 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.433354 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.433345 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.433524 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.433441 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.433524 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.433466 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h96v\" (UniqueName: \"kubernetes.io/projected/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-kube-api-access-9h96v\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.433524 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.433485 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.433524 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.433511 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.535191 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.535159 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.535375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.535224 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.535375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.535254 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h96v\" (UniqueName: \"kubernetes.io/projected/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-kube-api-access-9h96v\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.535375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.535283 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.535375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.535316 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.535607 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.535394 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.535607 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.535443 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.535868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.535842 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.535972 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.535870 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.535972 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.535873 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.535972 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.535891 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.536119 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.535974 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.536430 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.536404 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.536677 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.536655 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.537698 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.537679 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.538257 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.538238 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.547986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.547965 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h96v\" (UniqueName: \"kubernetes.io/projected/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-kube-api-access-9h96v\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.552021 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.552004 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.714158 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.714053 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:03.801355 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.801323 4910 generic.go:356] "Generic (PLEG): container finished" podID="ab24981d-834d-49d0-bfa7-eb64ff0ef2d4" containerID="4f759eb2ccece62f8ce8084cac067123e3a2a63fe10fa840a5695ec89b3190da" exitCode=0 Oct 31 19:31:03.801720 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:03.801400 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed47bff-kz7g9" event={"ID":"ab24981d-834d-49d0-bfa7-eb64ff0ef2d4","Type":"ContainerDied","Data":"4f759eb2ccece62f8ce8084cac067123e3a2a63fe10fa840a5695ec89b3190da"} Oct 31 19:31:04.003990 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:31:04.003955 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336cb7b6_2b37_4e93_9c0c_1a2c07e8ca52.slice/crio-bf5cd0c60e887a3fb7a55cc4dfb4466540132950b27da023e3866f9cfd367871 WatchSource:0}: Error finding container bf5cd0c60e887a3fb7a55cc4dfb4466540132950b27da023e3866f9cfd367871: Status 404 returned error can't find the container with id bf5cd0c60e887a3fb7a55cc4dfb4466540132950b27da023e3866f9cfd367871 Oct 31 19:31:04.004797 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:04.004768 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z"] Oct 31 19:31:04.007147 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:04.007109 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:31:04.007235 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:04.007193 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:31:04.007274 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:04.007234 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:31:04.810235 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:04.810191 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" event={"ID":"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52","Type":"ContainerStarted","Data":"9c01437dae8454927a1d8cf486034b36fbb6999e0bd0d8a8793e664be8078ca3"} Oct 31 19:31:04.810691 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:04.810257 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" event={"ID":"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52","Type":"ContainerStarted","Data":"bf5cd0c60e887a3fb7a55cc4dfb4466540132950b27da023e3866f9cfd367871"} Oct 31 19:31:04.860237 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:04.860171 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" podStartSLOduration=1.860152348 podStartE2EDuration="1.860152348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:04.841189877 +0000 UTC m=+699.204010250" watchObservedRunningTime="2026-10-31 19:31:04.860152348 +0000 UTC m=+699.222972720" Oct 31 19:31:05.057782 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:05.057749 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed47bff-kz7g9" Oct 31 19:31:05.158365 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:05.158272 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab24981d-834d-49d0-bfa7-eb64ff0ef2d4-kube-api-access-v59hv\" (UniqueName: \"kubernetes.io/projected/ab24981d-834d-49d0-bfa7-eb64ff0ef2d4-kube-api-access-v59hv\") pod \"ab24981d-834d-49d0-bfa7-eb64ff0ef2d4\" (UID: \"ab24981d-834d-49d0-bfa7-eb64ff0ef2d4\") " Oct 31 19:31:05.158514 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:05.158452 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab24981d-834d-49d0-bfa7-eb64ff0ef2d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab24981d-834d-49d0-bfa7-eb64ff0ef2d4-maas-api-openshift-ca\") pod \"ab24981d-834d-49d0-bfa7-eb64ff0ef2d4\" (UID: \"ab24981d-834d-49d0-bfa7-eb64ff0ef2d4\") " Oct 31 19:31:05.158656 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:05.158638 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab24981d-834d-49d0-bfa7-eb64ff0ef2d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab24981d-834d-49d0-bfa7-eb64ff0ef2d4-maas-api-xks-ca\") pod \"ab24981d-834d-49d0-bfa7-eb64ff0ef2d4\" (UID: \"ab24981d-834d-49d0-bfa7-eb64ff0ef2d4\") " Oct 31 19:31:05.158879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:05.158851 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab24981d-834d-49d0-bfa7-eb64ff0ef2d4-maas-api-openshift-ca" pod "ab24981d-834d-49d0-bfa7-eb64ff0ef2d4" (UID: "ab24981d-834d-49d0-bfa7-eb64ff0ef2d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:31:05.159019 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:05.159001 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab24981d-834d-49d0-bfa7-eb64ff0ef2d4-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:05.160499 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:05.160476 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab24981d-834d-49d0-bfa7-eb64ff0ef2d4-kube-api-access-v59hv" pod "ab24981d-834d-49d0-bfa7-eb64ff0ef2d4" (UID: "ab24981d-834d-49d0-bfa7-eb64ff0ef2d4"). InnerVolumeSpecName "kube-api-access-v59hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:31:05.161132 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:05.161116 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab24981d-834d-49d0-bfa7-eb64ff0ef2d4-maas-api-xks-ca" pod "ab24981d-834d-49d0-bfa7-eb64ff0ef2d4" (UID: "ab24981d-834d-49d0-bfa7-eb64ff0ef2d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:31:05.260707 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:05.260657 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab24981d-834d-49d0-bfa7-eb64ff0ef2d4-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:05.260707 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:05.260698 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v59hv\" (UniqueName: \"kubernetes.io/projected/ab24981d-834d-49d0-bfa7-eb64ff0ef2d4-kube-api-access-v59hv\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:05.715145 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:05.715114 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:05.716611 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:05.716589 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 19:31:05.716720 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:05.716629 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" podUID="336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 19:31:05.817499 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:05.817468 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed47bff-kz7g9" Oct 31 19:31:05.817499 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:05.817488 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fed47bff-kz7g9" event={"ID":"ab24981d-834d-49d0-bfa7-eb64ff0ef2d4","Type":"ContainerDied","Data":"1850c04adbad8812988a698f27c3deb235ce004318ee00cee75dd98c6ce14c19"} Oct 31 19:31:05.817982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:05.817530 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1850c04adbad8812988a698f27c3deb235ce004318ee00cee75dd98c6ce14c19" Oct 31 19:31:06.713596 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:06.713557 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b891f4f-vnc7m"] Oct 31 19:31:06.714181 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:06.714162 4910 cpu_manager.go:414] "containerMap: removing container" podUID="ab24981d-834d-49d0-bfa7-eb64ff0ef2d4" containerName="revoke-keys" Oct 31 19:31:06.714295 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:06.714182 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab24981d-834d-49d0-bfa7-eb64ff0ef2d4" containerName="revoke-keys" Oct 31 19:31:06.714352 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:06.714302 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab24981d-834d-49d0-bfa7-eb64ff0ef2d4" containerName="revoke-keys" Oct 31 19:31:06.714704 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:06.714681 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 19:31:06.714781 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:06.714728 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" podUID="336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 19:31:06.722936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:06.717903 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b891f4f-vnc7m" Oct 31 19:31:06.734418 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:06.734373 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:31:06.734644 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:06.734627 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:31:06.748935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:06.743080 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b891f4f-vnc7m"] Oct 31 19:31:06.778630 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:06.778600 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9869f04-02cf-4d2a-bd99-cf22f64910a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b891f4f-vnc7m\" (UID: \"c9869f04-02cf-4d2a-bd99-cf22f64910a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b891f4f-vnc7m" Oct 31 19:31:06.778782 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:06.778720 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgksj\" (UniqueName: \"kubernetes.io/projected/c9869f04-02cf-4d2a-bd99-cf22f64910a8-kube-api-access-cgksj\") pod \"maas-api-revoke-sub-2b891f4f-vnc7m\" (UID: \"c9869f04-02cf-4d2a-bd99-cf22f64910a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b891f4f-vnc7m" Oct 31 19:31:06.778782 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:06.778759 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9869f04-02cf-4d2a-bd99-cf22f64910a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b891f4f-vnc7m\" (UID: \"c9869f04-02cf-4d2a-bd99-cf22f64910a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b891f4f-vnc7m" Oct 31 19:31:06.880355 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:06.880317 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9869f04-02cf-4d2a-bd99-cf22f64910a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b891f4f-vnc7m\" (UID: \"c9869f04-02cf-4d2a-bd99-cf22f64910a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b891f4f-vnc7m" Oct 31 19:31:06.880894 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:06.880405 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgksj\" (UniqueName: \"kubernetes.io/projected/c9869f04-02cf-4d2a-bd99-cf22f64910a8-kube-api-access-cgksj\") pod \"maas-api-revoke-sub-2b891f4f-vnc7m\" (UID: \"c9869f04-02cf-4d2a-bd99-cf22f64910a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b891f4f-vnc7m" Oct 31 19:31:06.880894 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:06.880435 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9869f04-02cf-4d2a-bd99-cf22f64910a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b891f4f-vnc7m\" (UID: \"c9869f04-02cf-4d2a-bd99-cf22f64910a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b891f4f-vnc7m" Oct 31 19:31:06.881149 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:06.881128 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9869f04-02cf-4d2a-bd99-cf22f64910a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b891f4f-vnc7m\" (UID: \"c9869f04-02cf-4d2a-bd99-cf22f64910a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b891f4f-vnc7m" Oct 31 19:31:06.882748 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:06.882723 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9869f04-02cf-4d2a-bd99-cf22f64910a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b891f4f-vnc7m\" (UID: \"c9869f04-02cf-4d2a-bd99-cf22f64910a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b891f4f-vnc7m" Oct 31 19:31:06.893575 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:06.893543 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgksj\" (UniqueName: \"kubernetes.io/projected/c9869f04-02cf-4d2a-bd99-cf22f64910a8-kube-api-access-cgksj\") pod \"maas-api-revoke-sub-2b891f4f-vnc7m\" (UID: \"c9869f04-02cf-4d2a-bd99-cf22f64910a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b891f4f-vnc7m" Oct 31 19:31:07.075182 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:07.075142 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b891f4f-vnc7m" Oct 31 19:31:07.387242 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:07.387215 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b891f4f-vnc7m"] Oct 31 19:31:07.387370 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:31:07.387270 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9869f04_02cf_4d2a_bd99_cf22f64910a8.slice/crio-7ddb70997ee022fb133d72a1487f9739abf2dd104b58b512c6547c55783d857d WatchSource:0}: Error finding container 7ddb70997ee022fb133d72a1487f9739abf2dd104b58b512c6547c55783d857d: Status 404 returned error can't find the container with id 7ddb70997ee022fb133d72a1487f9739abf2dd104b58b512c6547c55783d857d Oct 31 19:31:07.715059 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:07.714976 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 19:31:07.715059 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:07.715030 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" podUID="336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 19:31:07.864453 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:07.864401 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b891f4f-vnc7m" event={"ID":"c9869f04-02cf-4d2a-bd99-cf22f64910a8","Type":"ContainerStarted","Data":"9c1d9375aed24c7ffe1937099069e0b7dd131de486e268f53099d456a7f3ccef"} Oct 31 19:31:07.864629 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:07.864471 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b891f4f-vnc7m" event={"ID":"c9869f04-02cf-4d2a-bd99-cf22f64910a8","Type":"ContainerStarted","Data":"7ddb70997ee022fb133d72a1487f9739abf2dd104b58b512c6547c55783d857d"} Oct 31 19:31:07.888207 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:07.888149 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b891f4f-vnc7m" podStartSLOduration=1.888129057 podStartE2EDuration="1.888129057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:07.885047629 +0000 UTC m=+702.247868001" watchObservedRunningTime="2026-10-31 19:31:07.888129057 +0000 UTC m=+702.250949429" Oct 31 19:31:08.714969 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:08.714851 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 19:31:08.714969 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:08.714943 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" podUID="336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 19:31:08.877374 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:08.877340 4910 generic.go:356] "Generic (PLEG): container finished" podID="c9869f04-02cf-4d2a-bd99-cf22f64910a8" containerID="9c1d9375aed24c7ffe1937099069e0b7dd131de486e268f53099d456a7f3ccef" exitCode=0 Oct 31 19:31:08.877696 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:08.877676 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b891f4f-vnc7m" event={"ID":"c9869f04-02cf-4d2a-bd99-cf22f64910a8","Type":"ContainerDied","Data":"9c1d9375aed24c7ffe1937099069e0b7dd131de486e268f53099d456a7f3ccef"} Oct 31 19:31:09.732932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:09.726391 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 19:31:09.732932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:09.726456 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" podUID="336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 19:31:10.098026 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:10.098000 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b891f4f-vnc7m" Oct 31 19:31:10.253191 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:10.253159 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9869f04-02cf-4d2a-bd99-cf22f64910a8-kube-api-access-cgksj\" (UniqueName: \"kubernetes.io/projected/c9869f04-02cf-4d2a-bd99-cf22f64910a8-kube-api-access-cgksj\") pod \"c9869f04-02cf-4d2a-bd99-cf22f64910a8\" (UID: \"c9869f04-02cf-4d2a-bd99-cf22f64910a8\") " Oct 31 19:31:10.253388 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:10.253261 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9869f04-02cf-4d2a-bd99-cf22f64910a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9869f04-02cf-4d2a-bd99-cf22f64910a8-maas-api-xks-ca\") pod \"c9869f04-02cf-4d2a-bd99-cf22f64910a8\" (UID: \"c9869f04-02cf-4d2a-bd99-cf22f64910a8\") " Oct 31 19:31:10.253453 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:10.253429 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9869f04-02cf-4d2a-bd99-cf22f64910a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9869f04-02cf-4d2a-bd99-cf22f64910a8-maas-api-openshift-ca\") pod \"c9869f04-02cf-4d2a-bd99-cf22f64910a8\" (UID: \"c9869f04-02cf-4d2a-bd99-cf22f64910a8\") " Oct 31 19:31:10.253751 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:10.253724 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9869f04-02cf-4d2a-bd99-cf22f64910a8-maas-api-openshift-ca" pod "c9869f04-02cf-4d2a-bd99-cf22f64910a8" (UID: "c9869f04-02cf-4d2a-bd99-cf22f64910a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:31:10.254052 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:10.253973 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9869f04-02cf-4d2a-bd99-cf22f64910a8-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:10.255733 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:10.255703 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9869f04-02cf-4d2a-bd99-cf22f64910a8-maas-api-xks-ca" pod "c9869f04-02cf-4d2a-bd99-cf22f64910a8" (UID: "c9869f04-02cf-4d2a-bd99-cf22f64910a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:31:10.257052 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:10.257027 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9869f04-02cf-4d2a-bd99-cf22f64910a8-kube-api-access-cgksj" pod "c9869f04-02cf-4d2a-bd99-cf22f64910a8" (UID: "c9869f04-02cf-4d2a-bd99-cf22f64910a8"). InnerVolumeSpecName "kube-api-access-cgksj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:31:10.355115 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:10.355023 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9869f04-02cf-4d2a-bd99-cf22f64910a8-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:10.355115 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:10.355069 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgksj\" (UniqueName: \"kubernetes.io/projected/c9869f04-02cf-4d2a-bd99-cf22f64910a8-kube-api-access-cgksj\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:10.715354 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:10.715266 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 19:31:10.715354 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:10.715337 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" podUID="336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 19:31:10.892214 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:10.892156 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b891f4f-vnc7m" event={"ID":"c9869f04-02cf-4d2a-bd99-cf22f64910a8","Type":"ContainerDied","Data":"7ddb70997ee022fb133d72a1487f9739abf2dd104b58b512c6547c55783d857d"} Oct 31 19:31:10.892747 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:10.892241 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ddb70997ee022fb133d72a1487f9739abf2dd104b58b512c6547c55783d857d" Oct 31 19:31:10.892747 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:10.892185 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b891f4f-vnc7m" Oct 31 19:31:11.719367 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:11.719333 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:11.719832 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:11.719809 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:11.720643 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:11.720622 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:31:12.047166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:12.047079 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f8d8368-dm6rd"] Oct 31 19:31:12.047630 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:12.047616 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c9869f04-02cf-4d2a-bd99-cf22f64910a8" containerName="revoke-keys" Oct 31 19:31:12.047630 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:12.047628 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9869f04-02cf-4d2a-bd99-cf22f64910a8" containerName="revoke-keys" Oct 31 19:31:12.047740 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:12.047715 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9869f04-02cf-4d2a-bd99-cf22f64910a8" containerName="revoke-keys" Oct 31 19:31:12.051197 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:12.051177 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8d8368-dm6rd" Oct 31 19:31:12.059247 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:12.059221 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:31:12.059554 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:12.059534 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:31:12.064582 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:12.064557 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f8d8368-dm6rd"] Oct 31 19:31:12.180760 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:12.180718 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f8d8368-dm6rd\" (UID: \"1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8d8368-dm6rd" Oct 31 19:31:12.180942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:12.180839 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sczx5\" (UniqueName: \"kubernetes.io/projected/1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec-kube-api-access-sczx5\") pod \"maas-api-revoke-sub-5f8d8368-dm6rd\" (UID: \"1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8d8368-dm6rd" Oct 31 19:31:12.181015 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:12.180939 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f8d8368-dm6rd\" (UID: \"1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8d8368-dm6rd" Oct 31 19:31:12.281624 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:12.281587 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f8d8368-dm6rd\" (UID: \"1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8d8368-dm6rd" Oct 31 19:31:12.281772 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:12.281642 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sczx5\" (UniqueName: \"kubernetes.io/projected/1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec-kube-api-access-sczx5\") pod \"maas-api-revoke-sub-5f8d8368-dm6rd\" (UID: \"1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8d8368-dm6rd" Oct 31 19:31:12.281772 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:12.281680 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f8d8368-dm6rd\" (UID: \"1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8d8368-dm6rd" Oct 31 19:31:12.282403 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:12.282377 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f8d8368-dm6rd\" (UID: \"1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8d8368-dm6rd" Oct 31 19:31:12.284032 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:12.284010 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f8d8368-dm6rd\" (UID: \"1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8d8368-dm6rd" Oct 31 19:31:12.294817 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:12.294790 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sczx5\" (UniqueName: \"kubernetes.io/projected/1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec-kube-api-access-sczx5\") pod \"maas-api-revoke-sub-5f8d8368-dm6rd\" (UID: \"1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8d8368-dm6rd" Oct 31 19:31:12.367539 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:12.367495 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8d8368-dm6rd" Oct 31 19:31:12.646382 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:12.646345 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f8d8368-dm6rd"] Oct 31 19:31:12.647202 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:31:12.647175 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d99da01_5cc4_44c4_8ba5_3b50abd0d2ec.slice/crio-54a86214c3bbc22eea24ad3a9253ec3c276e301f0108c2dcd71e4d7d5188457a WatchSource:0}: Error finding container 54a86214c3bbc22eea24ad3a9253ec3c276e301f0108c2dcd71e4d7d5188457a: Status 404 returned error can't find the container with id 54a86214c3bbc22eea24ad3a9253ec3c276e301f0108c2dcd71e4d7d5188457a Oct 31 19:31:12.714143 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:31:12.714106 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:31:12.906938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:12.906832 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8d8368-dm6rd" event={"ID":"1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec","Type":"ContainerStarted","Data":"54a86214c3bbc22eea24ad3a9253ec3c276e301f0108c2dcd71e4d7d5188457a"} Oct 31 19:31:13.916099 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:13.916055 4910 generic.go:356] "Generic (PLEG): container finished" podID="1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec" containerID="dae0a9ccec4f71bc359c483b8d6055f4de77c501a79f85b9620aadf258bb7ab5" exitCode=0 Oct 31 19:31:13.916533 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:13.916119 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8d8368-dm6rd" event={"ID":"1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec","Type":"ContainerDied","Data":"dae0a9ccec4f71bc359c483b8d6055f4de77c501a79f85b9620aadf258bb7ab5"} Oct 31 19:31:14.406443 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:14.406397 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96cb75ff-9msqn"] Oct 31 19:31:14.410297 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:14.410275 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cb75ff-9msqn" Oct 31 19:31:14.439454 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:14.437414 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96cb75ff-9msqn"] Oct 31 19:31:14.505299 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:14.505266 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqsgd\" (UniqueName: \"kubernetes.io/projected/a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7-kube-api-access-bqsgd\") pod \"maas-api-revoke-sub-96cb75ff-9msqn\" (UID: \"a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cb75ff-9msqn" Oct 31 19:31:14.505541 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:14.505331 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96cb75ff-9msqn\" (UID: \"a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cb75ff-9msqn" Oct 31 19:31:14.505541 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:14.505429 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96cb75ff-9msqn\" (UID: \"a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cb75ff-9msqn" Oct 31 19:31:14.606320 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:14.606284 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqsgd\" (UniqueName: \"kubernetes.io/projected/a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7-kube-api-access-bqsgd\") pod \"maas-api-revoke-sub-96cb75ff-9msqn\" (UID: \"a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cb75ff-9msqn" Oct 31 19:31:14.606489 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:14.606358 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96cb75ff-9msqn\" (UID: \"a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cb75ff-9msqn" Oct 31 19:31:14.606489 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:14.606380 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96cb75ff-9msqn\" (UID: \"a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cb75ff-9msqn" Oct 31 19:31:14.607071 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:14.607043 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96cb75ff-9msqn\" (UID: \"a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cb75ff-9msqn" Oct 31 19:31:14.608692 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:14.608672 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96cb75ff-9msqn\" (UID: \"a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cb75ff-9msqn" Oct 31 19:31:14.622083 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:14.622054 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqsgd\" (UniqueName: \"kubernetes.io/projected/a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7-kube-api-access-bqsgd\") pod \"maas-api-revoke-sub-96cb75ff-9msqn\" (UID: \"a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cb75ff-9msqn" Oct 31 19:31:14.724120 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:14.724015 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cb75ff-9msqn" Oct 31 19:31:15.040005 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.039976 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96cb75ff-9msqn"] Oct 31 19:31:15.093064 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.093040 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8d8368-dm6rd" Oct 31 19:31:15.219217 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.219188 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec-maas-api-openshift-ca\") pod \"1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec\" (UID: \"1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec\") " Oct 31 19:31:15.219390 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.219326 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec-maas-api-xks-ca\") pod \"1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec\" (UID: \"1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec\") " Oct 31 19:31:15.219445 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.219406 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec-kube-api-access-sczx5\" (UniqueName: \"kubernetes.io/projected/1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec-kube-api-access-sczx5\") pod \"1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec\" (UID: \"1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec\") " Oct 31 19:31:15.219600 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.219577 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec-maas-api-openshift-ca" pod "1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec" (UID: "1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:31:15.219977 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.219951 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:15.222116 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.222084 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec-kube-api-access-sczx5" pod "1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec" (UID: "1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec"). InnerVolumeSpecName "kube-api-access-sczx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:31:15.222221 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.222143 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec-maas-api-xks-ca" pod "1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec" (UID: "1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:31:15.321834 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.321804 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:15.321834 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.321835 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sczx5\" (UniqueName: \"kubernetes.io/projected/1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec-kube-api-access-sczx5\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:15.735770 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.735734 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9a85867-9nm5s"] Oct 31 19:31:15.736559 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.736540 4910 cpu_manager.go:414] "containerMap: removing container" podUID="1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec" containerName="revoke-keys" Oct 31 19:31:15.736619 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.736563 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec" containerName="revoke-keys" Oct 31 19:31:15.736715 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.736704 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec" containerName="revoke-keys" Oct 31 19:31:15.740612 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.740589 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a85867-9nm5s" Oct 31 19:31:15.761068 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.760703 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9a85867-9nm5s"] Oct 31 19:31:15.929086 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.929043 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dd5789d-652e-401f-9ab7-fe1bc7adc002-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9a85867-9nm5s\" (UID: \"3dd5789d-652e-401f-9ab7-fe1bc7adc002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a85867-9nm5s" Oct 31 19:31:15.929224 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.929140 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dd5789d-652e-401f-9ab7-fe1bc7adc002-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9a85867-9nm5s\" (UID: \"3dd5789d-652e-401f-9ab7-fe1bc7adc002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a85867-9nm5s" Oct 31 19:31:15.929270 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.929249 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdzsp\" (UniqueName: \"kubernetes.io/projected/3dd5789d-652e-401f-9ab7-fe1bc7adc002-kube-api-access-cdzsp\") pod \"maas-api-revoke-sub-d9a85867-9nm5s\" (UID: \"3dd5789d-652e-401f-9ab7-fe1bc7adc002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a85867-9nm5s" Oct 31 19:31:15.942448 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.942415 4910 generic.go:356] "Generic (PLEG): container finished" podID="a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7" containerID="cd87e8b20ba7e71c08fe6b9ee453ead947cb6423ecebcff5b07827b03566a537" exitCode=0 Oct 31 19:31:15.942585 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.942503 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cb75ff-9msqn" event={"ID":"a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7","Type":"ContainerDied","Data":"cd87e8b20ba7e71c08fe6b9ee453ead947cb6423ecebcff5b07827b03566a537"} Oct 31 19:31:15.942585 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.942569 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cb75ff-9msqn" event={"ID":"a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7","Type":"ContainerStarted","Data":"65b26f8814009e3b554b12f130f9e1280f0d949b0135f5fb68374e7283b72edb"} Oct 31 19:31:15.944120 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.944098 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8d8368-dm6rd" event={"ID":"1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec","Type":"ContainerDied","Data":"54a86214c3bbc22eea24ad3a9253ec3c276e301f0108c2dcd71e4d7d5188457a"} Oct 31 19:31:15.944231 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.944160 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54a86214c3bbc22eea24ad3a9253ec3c276e301f0108c2dcd71e4d7d5188457a" Oct 31 19:31:15.944231 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:15.944105 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f8d8368-dm6rd" Oct 31 19:31:16.029864 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:16.029835 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dd5789d-652e-401f-9ab7-fe1bc7adc002-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9a85867-9nm5s\" (UID: \"3dd5789d-652e-401f-9ab7-fe1bc7adc002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a85867-9nm5s" Oct 31 19:31:16.030071 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:16.029907 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dd5789d-652e-401f-9ab7-fe1bc7adc002-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9a85867-9nm5s\" (UID: \"3dd5789d-652e-401f-9ab7-fe1bc7adc002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a85867-9nm5s" Oct 31 19:31:16.030071 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:16.030026 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdzsp\" (UniqueName: \"kubernetes.io/projected/3dd5789d-652e-401f-9ab7-fe1bc7adc002-kube-api-access-cdzsp\") pod \"maas-api-revoke-sub-d9a85867-9nm5s\" (UID: \"3dd5789d-652e-401f-9ab7-fe1bc7adc002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a85867-9nm5s" Oct 31 19:31:16.030550 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:16.030525 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dd5789d-652e-401f-9ab7-fe1bc7adc002-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9a85867-9nm5s\" (UID: \"3dd5789d-652e-401f-9ab7-fe1bc7adc002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a85867-9nm5s" Oct 31 19:31:16.032232 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:16.032214 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dd5789d-652e-401f-9ab7-fe1bc7adc002-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9a85867-9nm5s\" (UID: \"3dd5789d-652e-401f-9ab7-fe1bc7adc002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a85867-9nm5s" Oct 31 19:31:16.042631 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:16.042602 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdzsp\" (UniqueName: \"kubernetes.io/projected/3dd5789d-652e-401f-9ab7-fe1bc7adc002-kube-api-access-cdzsp\") pod \"maas-api-revoke-sub-d9a85867-9nm5s\" (UID: \"3dd5789d-652e-401f-9ab7-fe1bc7adc002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a85867-9nm5s" Oct 31 19:31:16.061508 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:16.061482 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a85867-9nm5s" Oct 31 19:31:16.349475 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:16.349447 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9a85867-9nm5s"] Oct 31 19:31:16.350517 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:31:16.350491 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dd5789d_652e_401f_9ab7_fe1bc7adc002.slice/crio-b09087350afbe4f187df7dc2c5a6a299956c19c0afc729488f5807b88dcb79e8 WatchSource:0}: Error finding container b09087350afbe4f187df7dc2c5a6a299956c19c0afc729488f5807b88dcb79e8: Status 404 returned error can't find the container with id b09087350afbe4f187df7dc2c5a6a299956c19c0afc729488f5807b88dcb79e8 Oct 31 19:31:16.952605 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:16.952508 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a85867-9nm5s" event={"ID":"3dd5789d-652e-401f-9ab7-fe1bc7adc002","Type":"ContainerStarted","Data":"22121b1178f4795221390bf79204a19b9d9cde91df292af141720ffb4776815b"} Oct 31 19:31:16.952605 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:16.952557 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a85867-9nm5s" event={"ID":"3dd5789d-652e-401f-9ab7-fe1bc7adc002","Type":"ContainerStarted","Data":"b09087350afbe4f187df7dc2c5a6a299956c19c0afc729488f5807b88dcb79e8"} Oct 31 19:31:16.975147 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:16.975091 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a85867-9nm5s" podStartSLOduration=1.975074914 podStartE2EDuration="1.975074914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:16.971487598 +0000 UTC m=+711.334307969" watchObservedRunningTime="2026-10-31 19:31:16.975074914 +0000 UTC m=+711.337895298" Oct 31 19:31:17.010787 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.010750 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fa6423c-85cc8"] Oct 31 19:31:17.015219 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.015190 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa6423c-85cc8" Oct 31 19:31:17.036522 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.036418 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fa6423c-85cc8"] Oct 31 19:31:17.126377 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.126351 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cb75ff-9msqn" Oct 31 19:31:17.147762 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.147724 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa4000ef-20a4-4566-bbfb-32be1f61145c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fa6423c-85cc8\" (UID: \"fa4000ef-20a4-4566-bbfb-32be1f61145c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa6423c-85cc8" Oct 31 19:31:17.147929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.147838 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjnvp\" (UniqueName: \"kubernetes.io/projected/fa4000ef-20a4-4566-bbfb-32be1f61145c-kube-api-access-kjnvp\") pod \"maas-api-revoke-sub-1fa6423c-85cc8\" (UID: \"fa4000ef-20a4-4566-bbfb-32be1f61145c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa6423c-85cc8" Oct 31 19:31:17.148021 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.147945 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa4000ef-20a4-4566-bbfb-32be1f61145c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fa6423c-85cc8\" (UID: \"fa4000ef-20a4-4566-bbfb-32be1f61145c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa6423c-85cc8" Oct 31 19:31:17.248452 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.248425 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7-maas-api-xks-ca\") pod \"a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7\" (UID: \"a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7\") " Oct 31 19:31:17.248572 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.248475 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7-maas-api-openshift-ca\") pod \"a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7\" (UID: \"a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7\") " Oct 31 19:31:17.248670 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.248656 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7-kube-api-access-bqsgd\" (UniqueName: \"kubernetes.io/projected/a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7-kube-api-access-bqsgd\") pod \"a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7\" (UID: \"a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7\") " Oct 31 19:31:17.248870 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.248847 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa4000ef-20a4-4566-bbfb-32be1f61145c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fa6423c-85cc8\" (UID: \"fa4000ef-20a4-4566-bbfb-32be1f61145c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa6423c-85cc8" Oct 31 19:31:17.248977 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.248879 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7-maas-api-openshift-ca" pod "a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7" (UID: "a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:31:17.249118 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.249098 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa4000ef-20a4-4566-bbfb-32be1f61145c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fa6423c-85cc8\" (UID: \"fa4000ef-20a4-4566-bbfb-32be1f61145c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa6423c-85cc8" Oct 31 19:31:17.249657 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.249194 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjnvp\" (UniqueName: \"kubernetes.io/projected/fa4000ef-20a4-4566-bbfb-32be1f61145c-kube-api-access-kjnvp\") pod \"maas-api-revoke-sub-1fa6423c-85cc8\" (UID: \"fa4000ef-20a4-4566-bbfb-32be1f61145c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa6423c-85cc8" Oct 31 19:31:17.249657 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.249291 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:17.249657 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.249623 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa4000ef-20a4-4566-bbfb-32be1f61145c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fa6423c-85cc8\" (UID: \"fa4000ef-20a4-4566-bbfb-32be1f61145c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa6423c-85cc8" Oct 31 19:31:17.250980 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.250952 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7-kube-api-access-bqsgd" pod "a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7" (UID: "a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7"). InnerVolumeSpecName "kube-api-access-bqsgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:31:17.251074 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.250979 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7-maas-api-xks-ca" pod "a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7" (UID: "a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:31:17.251614 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.251592 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa4000ef-20a4-4566-bbfb-32be1f61145c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fa6423c-85cc8\" (UID: \"fa4000ef-20a4-4566-bbfb-32be1f61145c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa6423c-85cc8" Oct 31 19:31:17.264988 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.264966 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjnvp\" (UniqueName: \"kubernetes.io/projected/fa4000ef-20a4-4566-bbfb-32be1f61145c-kube-api-access-kjnvp\") pod \"maas-api-revoke-sub-1fa6423c-85cc8\" (UID: \"fa4000ef-20a4-4566-bbfb-32be1f61145c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa6423c-85cc8" Oct 31 19:31:17.345531 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.345500 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa6423c-85cc8" Oct 31 19:31:17.350121 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.350095 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:17.350121 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.350121 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqsgd\" (UniqueName: \"kubernetes.io/projected/a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7-kube-api-access-bqsgd\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:17.721674 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.721640 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fa6423c-85cc8"] Oct 31 19:31:17.722322 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:31:17.722298 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa4000ef_20a4_4566_bbfb_32be1f61145c.slice/crio-09ed78cc6d372f34f0b9be2eb018dd205df28ab84a78b2a1220da280e20d363a WatchSource:0}: Error finding container 09ed78cc6d372f34f0b9be2eb018dd205df28ab84a78b2a1220da280e20d363a: Status 404 returned error can't find the container with id 09ed78cc6d372f34f0b9be2eb018dd205df28ab84a78b2a1220da280e20d363a Oct 31 19:31:17.970710 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.970550 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa6423c-85cc8" event={"ID":"fa4000ef-20a4-4566-bbfb-32be1f61145c","Type":"ContainerStarted","Data":"09ed78cc6d372f34f0b9be2eb018dd205df28ab84a78b2a1220da280e20d363a"} Oct 31 19:31:17.976505 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.976467 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cb75ff-9msqn" event={"ID":"a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7","Type":"ContainerDied","Data":"65b26f8814009e3b554b12f130f9e1280f0d949b0135f5fb68374e7283b72edb"} Oct 31 19:31:17.976645 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.976531 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65b26f8814009e3b554b12f130f9e1280f0d949b0135f5fb68374e7283b72edb" Oct 31 19:31:17.977099 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.976491 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96cb75ff-9msqn" Oct 31 19:31:17.986630 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.986384 4910 generic.go:356] "Generic (PLEG): container finished" podID="3dd5789d-652e-401f-9ab7-fe1bc7adc002" containerID="22121b1178f4795221390bf79204a19b9d9cde91df292af141720ffb4776815b" exitCode=0 Oct 31 19:31:17.986630 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:17.986470 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a85867-9nm5s" event={"ID":"3dd5789d-652e-401f-9ab7-fe1bc7adc002","Type":"ContainerDied","Data":"22121b1178f4795221390bf79204a19b9d9cde91df292af141720ffb4776815b"} Oct 31 19:31:18.994431 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:18.994397 4910 generic.go:356] "Generic (PLEG): container finished" podID="fa4000ef-20a4-4566-bbfb-32be1f61145c" containerID="c197d11e536b593a302211b9fb17668a3e800375744fad71ca672c6eac8085f2" exitCode=0 Oct 31 19:31:18.994824 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:18.994475 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa6423c-85cc8" event={"ID":"fa4000ef-20a4-4566-bbfb-32be1f61145c","Type":"ContainerDied","Data":"c197d11e536b593a302211b9fb17668a3e800375744fad71ca672c6eac8085f2"} Oct 31 19:31:19.163163 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:19.163135 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a85867-9nm5s" Oct 31 19:31:19.185539 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:19.185510 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dd5789d-652e-401f-9ab7-fe1bc7adc002-kube-api-access-cdzsp\" (UniqueName: \"kubernetes.io/projected/3dd5789d-652e-401f-9ab7-fe1bc7adc002-kube-api-access-cdzsp\") pod \"3dd5789d-652e-401f-9ab7-fe1bc7adc002\" (UID: \"3dd5789d-652e-401f-9ab7-fe1bc7adc002\") " Oct 31 19:31:19.185682 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:19.185551 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dd5789d-652e-401f-9ab7-fe1bc7adc002-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dd5789d-652e-401f-9ab7-fe1bc7adc002-maas-api-openshift-ca\") pod \"3dd5789d-652e-401f-9ab7-fe1bc7adc002\" (UID: \"3dd5789d-652e-401f-9ab7-fe1bc7adc002\") " Oct 31 19:31:19.185743 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:19.185705 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dd5789d-652e-401f-9ab7-fe1bc7adc002-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dd5789d-652e-401f-9ab7-fe1bc7adc002-maas-api-xks-ca\") pod \"3dd5789d-652e-401f-9ab7-fe1bc7adc002\" (UID: \"3dd5789d-652e-401f-9ab7-fe1bc7adc002\") " Oct 31 19:31:19.185959 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:19.185937 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dd5789d-652e-401f-9ab7-fe1bc7adc002-maas-api-openshift-ca" pod "3dd5789d-652e-401f-9ab7-fe1bc7adc002" (UID: "3dd5789d-652e-401f-9ab7-fe1bc7adc002"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:31:19.186147 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:19.186130 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dd5789d-652e-401f-9ab7-fe1bc7adc002-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:19.187630 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:19.187600 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dd5789d-652e-401f-9ab7-fe1bc7adc002-kube-api-access-cdzsp" pod "3dd5789d-652e-401f-9ab7-fe1bc7adc002" (UID: "3dd5789d-652e-401f-9ab7-fe1bc7adc002"). InnerVolumeSpecName "kube-api-access-cdzsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:31:19.187630 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:19.187627 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dd5789d-652e-401f-9ab7-fe1bc7adc002-maas-api-xks-ca" pod "3dd5789d-652e-401f-9ab7-fe1bc7adc002" (UID: "3dd5789d-652e-401f-9ab7-fe1bc7adc002"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:31:19.287766 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:19.287731 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dd5789d-652e-401f-9ab7-fe1bc7adc002-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:19.287766 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:19.287761 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdzsp\" (UniqueName: \"kubernetes.io/projected/3dd5789d-652e-401f-9ab7-fe1bc7adc002-kube-api-access-cdzsp\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:20.002421 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:20.002378 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a85867-9nm5s" event={"ID":"3dd5789d-652e-401f-9ab7-fe1bc7adc002","Type":"ContainerDied","Data":"b09087350afbe4f187df7dc2c5a6a299956c19c0afc729488f5807b88dcb79e8"} Oct 31 19:31:20.002421 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:20.002406 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9a85867-9nm5s" Oct 31 19:31:20.002861 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:20.002445 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b09087350afbe4f187df7dc2c5a6a299956c19c0afc729488f5807b88dcb79e8" Oct 31 19:31:20.166137 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:20.166113 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa6423c-85cc8" Oct 31 19:31:20.197569 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:20.197536 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa4000ef-20a4-4566-bbfb-32be1f61145c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa4000ef-20a4-4566-bbfb-32be1f61145c-maas-api-openshift-ca\") pod \"fa4000ef-20a4-4566-bbfb-32be1f61145c\" (UID: \"fa4000ef-20a4-4566-bbfb-32be1f61145c\") " Oct 31 19:31:20.197712 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:20.197676 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa4000ef-20a4-4566-bbfb-32be1f61145c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa4000ef-20a4-4566-bbfb-32be1f61145c-maas-api-xks-ca\") pod \"fa4000ef-20a4-4566-bbfb-32be1f61145c\" (UID: \"fa4000ef-20a4-4566-bbfb-32be1f61145c\") " Oct 31 19:31:20.197795 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:20.197779 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa4000ef-20a4-4566-bbfb-32be1f61145c-kube-api-access-kjnvp\" (UniqueName: \"kubernetes.io/projected/fa4000ef-20a4-4566-bbfb-32be1f61145c-kube-api-access-kjnvp\") pod \"fa4000ef-20a4-4566-bbfb-32be1f61145c\" (UID: \"fa4000ef-20a4-4566-bbfb-32be1f61145c\") " Oct 31 19:31:20.197875 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:20.197851 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa4000ef-20a4-4566-bbfb-32be1f61145c-maas-api-openshift-ca" pod "fa4000ef-20a4-4566-bbfb-32be1f61145c" (UID: "fa4000ef-20a4-4566-bbfb-32be1f61145c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:31:20.198218 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:20.198195 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa4000ef-20a4-4566-bbfb-32be1f61145c-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:20.200086 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:20.200059 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa4000ef-20a4-4566-bbfb-32be1f61145c-maas-api-xks-ca" pod "fa4000ef-20a4-4566-bbfb-32be1f61145c" (UID: "fa4000ef-20a4-4566-bbfb-32be1f61145c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:31:20.200202 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:20.200179 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4000ef-20a4-4566-bbfb-32be1f61145c-kube-api-access-kjnvp" pod "fa4000ef-20a4-4566-bbfb-32be1f61145c" (UID: "fa4000ef-20a4-4566-bbfb-32be1f61145c"). InnerVolumeSpecName "kube-api-access-kjnvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:31:20.299461 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:20.299426 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa4000ef-20a4-4566-bbfb-32be1f61145c-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:20.299461 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:20.299456 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjnvp\" (UniqueName: \"kubernetes.io/projected/fa4000ef-20a4-4566-bbfb-32be1f61145c-kube-api-access-kjnvp\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:21.018709 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:21.018671 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa6423c-85cc8" Oct 31 19:31:21.018709 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:21.018688 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fa6423c-85cc8" event={"ID":"fa4000ef-20a4-4566-bbfb-32be1f61145c","Type":"ContainerDied","Data":"09ed78cc6d372f34f0b9be2eb018dd205df28ab84a78b2a1220da280e20d363a"} Oct 31 19:31:21.019283 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:21.018757 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09ed78cc6d372f34f0b9be2eb018dd205df28ab84a78b2a1220da280e20d363a" Oct 31 19:31:24.482408 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:24.482378 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98dd5c4e-xg769"] Oct 31 19:31:24.482886 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:24.482872 4910 cpu_manager.go:414] "containerMap: removing container" podUID="a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7" containerName="revoke-keys" Oct 31 19:31:24.482947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:24.482887 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7" containerName="revoke-keys" Oct 31 19:31:24.482947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:24.482906 4910 cpu_manager.go:414] "containerMap: removing container" podUID="3dd5789d-652e-401f-9ab7-fe1bc7adc002" containerName="revoke-keys" Oct 31 19:31:24.482947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:24.482925 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dd5789d-652e-401f-9ab7-fe1bc7adc002" containerName="revoke-keys" Oct 31 19:31:24.482947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:24.482945 4910 cpu_manager.go:414] "containerMap: removing container" podUID="fa4000ef-20a4-4566-bbfb-32be1f61145c" containerName="revoke-keys" Oct 31 19:31:24.483071 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:24.482949 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa4000ef-20a4-4566-bbfb-32be1f61145c" containerName="revoke-keys" Oct 31 19:31:24.483071 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:24.483038 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7" containerName="revoke-keys" Oct 31 19:31:24.483071 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:24.483049 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dd5789d-652e-401f-9ab7-fe1bc7adc002" containerName="revoke-keys" Oct 31 19:31:24.483071 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:24.483063 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa4000ef-20a4-4566-bbfb-32be1f61145c" containerName="revoke-keys" Oct 31 19:31:24.487755 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:24.487734 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dd5c4e-xg769" Oct 31 19:31:24.498015 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:24.497987 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:31:24.499046 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:24.498956 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:31:24.502334 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:24.502304 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98dd5c4e-xg769"] Oct 31 19:31:24.562421 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:24.562382 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b05a049b-9fe7-4ee0-b007-4149edb2da92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98dd5c4e-xg769\" (UID: \"b05a049b-9fe7-4ee0-b007-4149edb2da92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dd5c4e-xg769" Oct 31 19:31:24.562618 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:24.562544 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b05a049b-9fe7-4ee0-b007-4149edb2da92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98dd5c4e-xg769\" (UID: \"b05a049b-9fe7-4ee0-b007-4149edb2da92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dd5c4e-xg769" Oct 31 19:31:24.562706 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:24.562679 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpt5s\" (UniqueName: \"kubernetes.io/projected/b05a049b-9fe7-4ee0-b007-4149edb2da92-kube-api-access-gpt5s\") pod \"maas-api-revoke-sub-98dd5c4e-xg769\" (UID: \"b05a049b-9fe7-4ee0-b007-4149edb2da92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dd5c4e-xg769" Oct 31 19:31:24.664179 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:24.664118 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b05a049b-9fe7-4ee0-b007-4149edb2da92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98dd5c4e-xg769\" (UID: \"b05a049b-9fe7-4ee0-b007-4149edb2da92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dd5c4e-xg769" Oct 31 19:31:24.664384 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:24.664320 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b05a049b-9fe7-4ee0-b007-4149edb2da92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98dd5c4e-xg769\" (UID: \"b05a049b-9fe7-4ee0-b007-4149edb2da92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dd5c4e-xg769" Oct 31 19:31:24.664470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:24.664414 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpt5s\" (UniqueName: \"kubernetes.io/projected/b05a049b-9fe7-4ee0-b007-4149edb2da92-kube-api-access-gpt5s\") pod \"maas-api-revoke-sub-98dd5c4e-xg769\" (UID: \"b05a049b-9fe7-4ee0-b007-4149edb2da92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dd5c4e-xg769" Oct 31 19:31:24.665753 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:24.665723 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b05a049b-9fe7-4ee0-b007-4149edb2da92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98dd5c4e-xg769\" (UID: \"b05a049b-9fe7-4ee0-b007-4149edb2da92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dd5c4e-xg769" Oct 31 19:31:24.667493 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:24.667467 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b05a049b-9fe7-4ee0-b007-4149edb2da92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98dd5c4e-xg769\" (UID: \"b05a049b-9fe7-4ee0-b007-4149edb2da92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dd5c4e-xg769" Oct 31 19:31:24.691885 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:24.691839 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpt5s\" (UniqueName: \"kubernetes.io/projected/b05a049b-9fe7-4ee0-b007-4149edb2da92-kube-api-access-gpt5s\") pod \"maas-api-revoke-sub-98dd5c4e-xg769\" (UID: \"b05a049b-9fe7-4ee0-b007-4149edb2da92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dd5c4e-xg769" Oct 31 19:31:24.802008 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:24.801927 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dd5c4e-xg769" Oct 31 19:31:25.100950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:25.094812 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98dd5c4e-xg769"] Oct 31 19:31:25.100950 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:31:25.100698 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05a049b_9fe7_4ee0_b007_4149edb2da92.slice/crio-caf93450581bedbe71cdb19abcc6bd4c586090419617f736313e1c9ec6d83c5b WatchSource:0}: Error finding container caf93450581bedbe71cdb19abcc6bd4c586090419617f736313e1c9ec6d83c5b: Status 404 returned error can't find the container with id caf93450581bedbe71cdb19abcc6bd4c586090419617f736313e1c9ec6d83c5b Oct 31 19:31:26.053869 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:26.053818 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dd5c4e-xg769" event={"ID":"b05a049b-9fe7-4ee0-b007-4149edb2da92","Type":"ContainerStarted","Data":"4316831fea2a129e74ad1748aecb5388ccd9d60b314f9654c41aa4292e25419a"} Oct 31 19:31:26.054289 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:26.053886 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dd5c4e-xg769" event={"ID":"b05a049b-9fe7-4ee0-b007-4149edb2da92","Type":"ContainerStarted","Data":"caf93450581bedbe71cdb19abcc6bd4c586090419617f736313e1c9ec6d83c5b"} Oct 31 19:31:26.088211 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:26.088152 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dd5c4e-xg769" podStartSLOduration=2.088129583 podStartE2EDuration="2.088129583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:26.086623298 +0000 UTC m=+720.449443671" watchObservedRunningTime="2026-10-31 19:31:26.088129583 +0000 UTC m=+720.450949995" Oct 31 19:31:27.064011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:27.063975 4910 generic.go:356] "Generic (PLEG): container finished" podID="b05a049b-9fe7-4ee0-b007-4149edb2da92" containerID="4316831fea2a129e74ad1748aecb5388ccd9d60b314f9654c41aa4292e25419a" exitCode=0 Oct 31 19:31:27.064400 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:27.064022 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dd5c4e-xg769" event={"ID":"b05a049b-9fe7-4ee0-b007-4149edb2da92","Type":"ContainerDied","Data":"4316831fea2a129e74ad1748aecb5388ccd9d60b314f9654c41aa4292e25419a"} Oct 31 19:31:27.780363 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:27.780325 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8ed7d9a-4f6bk"] Oct 31 19:31:27.806223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:27.806187 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8ed7d9a-4f6bk"] Oct 31 19:31:27.806377 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:27.806277 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ed7d9a-4f6bk" Oct 31 19:31:27.915190 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:27.915158 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86tgc\" (UniqueName: \"kubernetes.io/projected/2242e60d-89ec-4e15-b211-9781a2e1d0a6-kube-api-access-86tgc\") pod \"maas-api-revoke-sub-f8ed7d9a-4f6bk\" (UID: \"2242e60d-89ec-4e15-b211-9781a2e1d0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ed7d9a-4f6bk" Oct 31 19:31:27.915380 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:27.915336 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2242e60d-89ec-4e15-b211-9781a2e1d0a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8ed7d9a-4f6bk\" (UID: \"2242e60d-89ec-4e15-b211-9781a2e1d0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ed7d9a-4f6bk" Oct 31 19:31:27.915444 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:27.915393 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2242e60d-89ec-4e15-b211-9781a2e1d0a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8ed7d9a-4f6bk\" (UID: \"2242e60d-89ec-4e15-b211-9781a2e1d0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ed7d9a-4f6bk" Oct 31 19:31:28.016510 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:28.016472 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2242e60d-89ec-4e15-b211-9781a2e1d0a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8ed7d9a-4f6bk\" (UID: \"2242e60d-89ec-4e15-b211-9781a2e1d0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ed7d9a-4f6bk" Oct 31 19:31:28.016751 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:28.016551 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2242e60d-89ec-4e15-b211-9781a2e1d0a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8ed7d9a-4f6bk\" (UID: \"2242e60d-89ec-4e15-b211-9781a2e1d0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ed7d9a-4f6bk" Oct 31 19:31:28.016751 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:28.016616 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86tgc\" (UniqueName: \"kubernetes.io/projected/2242e60d-89ec-4e15-b211-9781a2e1d0a6-kube-api-access-86tgc\") pod \"maas-api-revoke-sub-f8ed7d9a-4f6bk\" (UID: \"2242e60d-89ec-4e15-b211-9781a2e1d0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ed7d9a-4f6bk" Oct 31 19:31:28.017131 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:28.017110 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2242e60d-89ec-4e15-b211-9781a2e1d0a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8ed7d9a-4f6bk\" (UID: \"2242e60d-89ec-4e15-b211-9781a2e1d0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ed7d9a-4f6bk" Oct 31 19:31:28.018973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:28.018947 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2242e60d-89ec-4e15-b211-9781a2e1d0a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8ed7d9a-4f6bk\" (UID: \"2242e60d-89ec-4e15-b211-9781a2e1d0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ed7d9a-4f6bk" Oct 31 19:31:28.031342 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:28.031282 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86tgc\" (UniqueName: \"kubernetes.io/projected/2242e60d-89ec-4e15-b211-9781a2e1d0a6-kube-api-access-86tgc\") pod \"maas-api-revoke-sub-f8ed7d9a-4f6bk\" (UID: \"2242e60d-89ec-4e15-b211-9781a2e1d0a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ed7d9a-4f6bk" Oct 31 19:31:28.124426 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:28.124385 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ed7d9a-4f6bk" Oct 31 19:31:28.242428 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:28.242097 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dd5c4e-xg769" Oct 31 19:31:28.321450 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:28.321418 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b05a049b-9fe7-4ee0-b007-4149edb2da92-kube-api-access-gpt5s\" (UniqueName: \"kubernetes.io/projected/b05a049b-9fe7-4ee0-b007-4149edb2da92-kube-api-access-gpt5s\") pod \"b05a049b-9fe7-4ee0-b007-4149edb2da92\" (UID: \"b05a049b-9fe7-4ee0-b007-4149edb2da92\") " Oct 31 19:31:28.321658 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:28.321561 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b05a049b-9fe7-4ee0-b007-4149edb2da92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b05a049b-9fe7-4ee0-b007-4149edb2da92-maas-api-xks-ca\") pod \"b05a049b-9fe7-4ee0-b007-4149edb2da92\" (UID: \"b05a049b-9fe7-4ee0-b007-4149edb2da92\") " Oct 31 19:31:28.321658 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:28.321620 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b05a049b-9fe7-4ee0-b007-4149edb2da92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b05a049b-9fe7-4ee0-b007-4149edb2da92-maas-api-openshift-ca\") pod \"b05a049b-9fe7-4ee0-b007-4149edb2da92\" (UID: \"b05a049b-9fe7-4ee0-b007-4149edb2da92\") " Oct 31 19:31:28.322066 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:28.322038 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b05a049b-9fe7-4ee0-b007-4149edb2da92-maas-api-openshift-ca" pod "b05a049b-9fe7-4ee0-b007-4149edb2da92" (UID: "b05a049b-9fe7-4ee0-b007-4149edb2da92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:31:28.323621 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:28.323588 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b05a049b-9fe7-4ee0-b007-4149edb2da92-kube-api-access-gpt5s" pod "b05a049b-9fe7-4ee0-b007-4149edb2da92" (UID: "b05a049b-9fe7-4ee0-b007-4149edb2da92"). InnerVolumeSpecName "kube-api-access-gpt5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:31:28.323984 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:28.323962 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b05a049b-9fe7-4ee0-b007-4149edb2da92-maas-api-xks-ca" pod "b05a049b-9fe7-4ee0-b007-4149edb2da92" (UID: "b05a049b-9fe7-4ee0-b007-4149edb2da92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:31:28.424024 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:28.423984 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b05a049b-9fe7-4ee0-b007-4149edb2da92-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:28.424024 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:28.424021 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b05a049b-9fe7-4ee0-b007-4149edb2da92-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:28.424267 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:28.424035 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpt5s\" (UniqueName: \"kubernetes.io/projected/b05a049b-9fe7-4ee0-b007-4149edb2da92-kube-api-access-gpt5s\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:28.429025 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:28.428996 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8ed7d9a-4f6bk"] Oct 31 19:31:28.430351 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:31:28.430316 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2242e60d_89ec_4e15_b211_9781a2e1d0a6.slice/crio-6fe387d59e101c6dad4fb3f9b07cdd8850ae54c5679b2d82e061ff29d363b024 WatchSource:0}: Error finding container 6fe387d59e101c6dad4fb3f9b07cdd8850ae54c5679b2d82e061ff29d363b024: Status 404 returned error can't find the container with id 6fe387d59e101c6dad4fb3f9b07cdd8850ae54c5679b2d82e061ff29d363b024 Oct 31 19:31:29.084525 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:29.084484 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ed7d9a-4f6bk" event={"ID":"2242e60d-89ec-4e15-b211-9781a2e1d0a6","Type":"ContainerStarted","Data":"1f95e9e0cc67b0d5acaa5ad4fa07ca0f6fb00e3a121512297ebd19eb8b273f1d"} Oct 31 19:31:29.084739 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:29.084547 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ed7d9a-4f6bk" event={"ID":"2242e60d-89ec-4e15-b211-9781a2e1d0a6","Type":"ContainerStarted","Data":"6fe387d59e101c6dad4fb3f9b07cdd8850ae54c5679b2d82e061ff29d363b024"} Oct 31 19:31:29.086049 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:29.086027 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dd5c4e-xg769" Oct 31 19:31:29.086168 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:29.086072 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dd5c4e-xg769" event={"ID":"b05a049b-9fe7-4ee0-b007-4149edb2da92","Type":"ContainerDied","Data":"caf93450581bedbe71cdb19abcc6bd4c586090419617f736313e1c9ec6d83c5b"} Oct 31 19:31:29.086168 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:29.086121 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="caf93450581bedbe71cdb19abcc6bd4c586090419617f736313e1c9ec6d83c5b" Oct 31 19:31:29.112374 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:29.112329 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ed7d9a-4f6bk" podStartSLOduration=2.112317301 podStartE2EDuration="2.112317301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:29.11072521 +0000 UTC m=+723.473545581" watchObservedRunningTime="2026-10-31 19:31:29.112317301 +0000 UTC m=+723.475137672" Oct 31 19:31:30.093815 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:30.093784 4910 generic.go:356] "Generic (PLEG): container finished" podID="2242e60d-89ec-4e15-b211-9781a2e1d0a6" containerID="1f95e9e0cc67b0d5acaa5ad4fa07ca0f6fb00e3a121512297ebd19eb8b273f1d" exitCode=0 Oct 31 19:31:30.094221 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:30.093866 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ed7d9a-4f6bk" event={"ID":"2242e60d-89ec-4e15-b211-9781a2e1d0a6","Type":"ContainerDied","Data":"1f95e9e0cc67b0d5acaa5ad4fa07ca0f6fb00e3a121512297ebd19eb8b273f1d"} Oct 31 19:31:31.921452 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:31.921133 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ed7d9a-4f6bk" Oct 31 19:31:32.030958 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.030880 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2242e60d-89ec-4e15-b211-9781a2e1d0a6-kube-api-access-86tgc\" (UniqueName: \"kubernetes.io/projected/2242e60d-89ec-4e15-b211-9781a2e1d0a6-kube-api-access-86tgc\") pod \"2242e60d-89ec-4e15-b211-9781a2e1d0a6\" (UID: \"2242e60d-89ec-4e15-b211-9781a2e1d0a6\") " Oct 31 19:31:32.031514 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.031404 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2242e60d-89ec-4e15-b211-9781a2e1d0a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2242e60d-89ec-4e15-b211-9781a2e1d0a6-maas-api-xks-ca\") pod \"2242e60d-89ec-4e15-b211-9781a2e1d0a6\" (UID: \"2242e60d-89ec-4e15-b211-9781a2e1d0a6\") " Oct 31 19:31:32.031681 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.031661 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2242e60d-89ec-4e15-b211-9781a2e1d0a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2242e60d-89ec-4e15-b211-9781a2e1d0a6-maas-api-openshift-ca\") pod \"2242e60d-89ec-4e15-b211-9781a2e1d0a6\" (UID: \"2242e60d-89ec-4e15-b211-9781a2e1d0a6\") " Oct 31 19:31:32.032778 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.032754 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2242e60d-89ec-4e15-b211-9781a2e1d0a6-maas-api-openshift-ca" pod "2242e60d-89ec-4e15-b211-9781a2e1d0a6" (UID: "2242e60d-89ec-4e15-b211-9781a2e1d0a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:31:32.073255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.073097 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2242e60d-89ec-4e15-b211-9781a2e1d0a6-maas-api-xks-ca" pod "2242e60d-89ec-4e15-b211-9781a2e1d0a6" (UID: "2242e60d-89ec-4e15-b211-9781a2e1d0a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:31:32.073255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.073224 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2242e60d-89ec-4e15-b211-9781a2e1d0a6-kube-api-access-86tgc" pod "2242e60d-89ec-4e15-b211-9781a2e1d0a6" (UID: "2242e60d-89ec-4e15-b211-9781a2e1d0a6"). InnerVolumeSpecName "kube-api-access-86tgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:31:32.131948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.131883 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ed7d9a-4f6bk" event={"ID":"2242e60d-89ec-4e15-b211-9781a2e1d0a6","Type":"ContainerDied","Data":"6fe387d59e101c6dad4fb3f9b07cdd8850ae54c5679b2d82e061ff29d363b024"} Oct 31 19:31:32.132463 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.132442 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fe387d59e101c6dad4fb3f9b07cdd8850ae54c5679b2d82e061ff29d363b024" Oct 31 19:31:32.132636 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.132263 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ed7d9a-4f6bk" Oct 31 19:31:32.135086 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.135066 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86tgc\" (UniqueName: \"kubernetes.io/projected/2242e60d-89ec-4e15-b211-9781a2e1d0a6-kube-api-access-86tgc\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:32.135256 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.135239 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2242e60d-89ec-4e15-b211-9781a2e1d0a6-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:32.135472 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.135456 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2242e60d-89ec-4e15-b211-9781a2e1d0a6-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:32.207124 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.207092 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7876437-hftq9"] Oct 31 19:31:32.207798 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.207777 4910 cpu_manager.go:414] "containerMap: removing container" podUID="b05a049b-9fe7-4ee0-b007-4149edb2da92" containerName="revoke-keys" Oct 31 19:31:32.207798 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.207796 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b05a049b-9fe7-4ee0-b007-4149edb2da92" containerName="revoke-keys" Oct 31 19:31:32.207990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.207844 4910 cpu_manager.go:414] "containerMap: removing container" podUID="2242e60d-89ec-4e15-b211-9781a2e1d0a6" containerName="revoke-keys" Oct 31 19:31:32.207990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.207852 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2242e60d-89ec-4e15-b211-9781a2e1d0a6" containerName="revoke-keys" Oct 31 19:31:32.207990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.207979 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="b05a049b-9fe7-4ee0-b007-4149edb2da92" containerName="revoke-keys" Oct 31 19:31:32.208085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.207998 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="2242e60d-89ec-4e15-b211-9781a2e1d0a6" containerName="revoke-keys" Oct 31 19:31:32.211774 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.211755 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7876437-hftq9" Oct 31 19:31:32.231143 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.223250 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:31:32.231143 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.224015 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:31:32.256936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.255961 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7876437-hftq9"] Oct 31 19:31:32.337957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.337898 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd8c8\" (UniqueName: \"kubernetes.io/projected/100a9e32-a903-4d0d-aed2-e4857230bd7d-kube-api-access-dd8c8\") pod \"maas-api-revoke-sub-d7876437-hftq9\" (UID: \"100a9e32-a903-4d0d-aed2-e4857230bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7876437-hftq9" Oct 31 19:31:32.338163 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.338064 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/100a9e32-a903-4d0d-aed2-e4857230bd7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7876437-hftq9\" (UID: \"100a9e32-a903-4d0d-aed2-e4857230bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7876437-hftq9" Oct 31 19:31:32.338163 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.338135 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/100a9e32-a903-4d0d-aed2-e4857230bd7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7876437-hftq9\" (UID: \"100a9e32-a903-4d0d-aed2-e4857230bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7876437-hftq9" Oct 31 19:31:32.439724 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.439681 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd8c8\" (UniqueName: \"kubernetes.io/projected/100a9e32-a903-4d0d-aed2-e4857230bd7d-kube-api-access-dd8c8\") pod \"maas-api-revoke-sub-d7876437-hftq9\" (UID: \"100a9e32-a903-4d0d-aed2-e4857230bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7876437-hftq9" Oct 31 19:31:32.439898 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.439837 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/100a9e32-a903-4d0d-aed2-e4857230bd7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7876437-hftq9\" (UID: \"100a9e32-a903-4d0d-aed2-e4857230bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7876437-hftq9" Oct 31 19:31:32.439898 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.439883 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/100a9e32-a903-4d0d-aed2-e4857230bd7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7876437-hftq9\" (UID: \"100a9e32-a903-4d0d-aed2-e4857230bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7876437-hftq9" Oct 31 19:31:32.440682 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.440659 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/100a9e32-a903-4d0d-aed2-e4857230bd7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7876437-hftq9\" (UID: \"100a9e32-a903-4d0d-aed2-e4857230bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7876437-hftq9" Oct 31 19:31:32.442482 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.442450 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/100a9e32-a903-4d0d-aed2-e4857230bd7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7876437-hftq9\" (UID: \"100a9e32-a903-4d0d-aed2-e4857230bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7876437-hftq9" Oct 31 19:31:32.462234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.462202 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd8c8\" (UniqueName: \"kubernetes.io/projected/100a9e32-a903-4d0d-aed2-e4857230bd7d-kube-api-access-dd8c8\") pod \"maas-api-revoke-sub-d7876437-hftq9\" (UID: \"100a9e32-a903-4d0d-aed2-e4857230bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7876437-hftq9" Oct 31 19:31:32.548353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.548314 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7876437-hftq9" Oct 31 19:31:32.848296 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:31:32.847059 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100a9e32_a903_4d0d_aed2_e4857230bd7d.slice/crio-ab34882df813c4fa70877b07c5f674a12a8e3338b053806c148648371f80b110 WatchSource:0}: Error finding container ab34882df813c4fa70877b07c5f674a12a8e3338b053806c148648371f80b110: Status 404 returned error can't find the container with id ab34882df813c4fa70877b07c5f674a12a8e3338b053806c148648371f80b110 Oct 31 19:31:32.856661 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:32.856634 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7876437-hftq9"] Oct 31 19:31:33.140191 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:33.140154 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7876437-hftq9" event={"ID":"100a9e32-a903-4d0d-aed2-e4857230bd7d","Type":"ContainerStarted","Data":"ab34882df813c4fa70877b07c5f674a12a8e3338b053806c148648371f80b110"} Oct 31 19:31:34.151495 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:34.151398 4910 generic.go:356] "Generic (PLEG): container finished" podID="100a9e32-a903-4d0d-aed2-e4857230bd7d" containerID="ff9736032812c377a7579c6852aed0b9aa2bc88556c2501841c7516e8a85f0de" exitCode=0 Oct 31 19:31:34.151992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:34.151537 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7876437-hftq9" event={"ID":"100a9e32-a903-4d0d-aed2-e4857230bd7d","Type":"ContainerDied","Data":"ff9736032812c377a7579c6852aed0b9aa2bc88556c2501841c7516e8a85f0de"} Oct 31 19:31:35.305301 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:35.305276 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7876437-hftq9" Oct 31 19:31:35.377560 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:35.377526 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/100a9e32-a903-4d0d-aed2-e4857230bd7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/100a9e32-a903-4d0d-aed2-e4857230bd7d-maas-api-openshift-ca\") pod \"100a9e32-a903-4d0d-aed2-e4857230bd7d\" (UID: \"100a9e32-a903-4d0d-aed2-e4857230bd7d\") " Oct 31 19:31:35.377732 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:35.377575 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/100a9e32-a903-4d0d-aed2-e4857230bd7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/100a9e32-a903-4d0d-aed2-e4857230bd7d-maas-api-xks-ca\") pod \"100a9e32-a903-4d0d-aed2-e4857230bd7d\" (UID: \"100a9e32-a903-4d0d-aed2-e4857230bd7d\") " Oct 31 19:31:35.377858 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:35.377839 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/100a9e32-a903-4d0d-aed2-e4857230bd7d-kube-api-access-dd8c8\" (UniqueName: \"kubernetes.io/projected/100a9e32-a903-4d0d-aed2-e4857230bd7d-kube-api-access-dd8c8\") pod \"100a9e32-a903-4d0d-aed2-e4857230bd7d\" (UID: \"100a9e32-a903-4d0d-aed2-e4857230bd7d\") " Oct 31 19:31:35.378005 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:35.377973 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/100a9e32-a903-4d0d-aed2-e4857230bd7d-maas-api-openshift-ca" pod "100a9e32-a903-4d0d-aed2-e4857230bd7d" (UID: "100a9e32-a903-4d0d-aed2-e4857230bd7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:31:35.378255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:35.378235 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/100a9e32-a903-4d0d-aed2-e4857230bd7d-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:35.384075 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:35.384047 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/100a9e32-a903-4d0d-aed2-e4857230bd7d-maas-api-xks-ca" pod "100a9e32-a903-4d0d-aed2-e4857230bd7d" (UID: "100a9e32-a903-4d0d-aed2-e4857230bd7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:31:35.384075 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:35.384053 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/100a9e32-a903-4d0d-aed2-e4857230bd7d-kube-api-access-dd8c8" pod "100a9e32-a903-4d0d-aed2-e4857230bd7d" (UID: "100a9e32-a903-4d0d-aed2-e4857230bd7d"). InnerVolumeSpecName "kube-api-access-dd8c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:31:35.479363 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:35.479283 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/100a9e32-a903-4d0d-aed2-e4857230bd7d-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:35.479363 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:35.479314 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd8c8\" (UniqueName: \"kubernetes.io/projected/100a9e32-a903-4d0d-aed2-e4857230bd7d-kube-api-access-dd8c8\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:36.167304 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:36.167277 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7876437-hftq9" Oct 31 19:31:36.167468 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:36.167309 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7876437-hftq9" event={"ID":"100a9e32-a903-4d0d-aed2-e4857230bd7d","Type":"ContainerDied","Data":"ab34882df813c4fa70877b07c5f674a12a8e3338b053806c148648371f80b110"} Oct 31 19:31:36.167468 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:36.167359 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab34882df813c4fa70877b07c5f674a12a8e3338b053806c148648371f80b110" Oct 31 19:31:38.035004 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:38.034966 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-221d6a18-n42bz"] Oct 31 19:31:38.035528 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:38.035513 4910 cpu_manager.go:414] "containerMap: removing container" podUID="100a9e32-a903-4d0d-aed2-e4857230bd7d" containerName="revoke-keys" Oct 31 19:31:38.035573 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:38.035530 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="100a9e32-a903-4d0d-aed2-e4857230bd7d" containerName="revoke-keys" Oct 31 19:31:38.035624 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:38.035615 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="100a9e32-a903-4d0d-aed2-e4857230bd7d" containerName="revoke-keys" Oct 31 19:31:38.039378 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:38.039361 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d6a18-n42bz" Oct 31 19:31:38.044720 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:38.044696 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:31:38.046470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:38.046448 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:31:38.052899 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:38.052874 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-221d6a18-n42bz"] Oct 31 19:31:38.111244 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:38.111214 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8drs\" (UniqueName: \"kubernetes.io/projected/3f85466a-6e79-4de7-b5e8-c33d02f708c4-kube-api-access-w8drs\") pod \"maas-api-revoke-sub-221d6a18-n42bz\" (UID: \"3f85466a-6e79-4de7-b5e8-c33d02f708c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d6a18-n42bz" Oct 31 19:31:38.111422 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:38.111282 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f85466a-6e79-4de7-b5e8-c33d02f708c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-221d6a18-n42bz\" (UID: \"3f85466a-6e79-4de7-b5e8-c33d02f708c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d6a18-n42bz" Oct 31 19:31:38.111422 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:38.111375 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f85466a-6e79-4de7-b5e8-c33d02f708c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-221d6a18-n42bz\" (UID: \"3f85466a-6e79-4de7-b5e8-c33d02f708c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d6a18-n42bz" Oct 31 19:31:38.212732 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:38.212691 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8drs\" (UniqueName: \"kubernetes.io/projected/3f85466a-6e79-4de7-b5e8-c33d02f708c4-kube-api-access-w8drs\") pod \"maas-api-revoke-sub-221d6a18-n42bz\" (UID: \"3f85466a-6e79-4de7-b5e8-c33d02f708c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d6a18-n42bz" Oct 31 19:31:38.213010 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:38.212800 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f85466a-6e79-4de7-b5e8-c33d02f708c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-221d6a18-n42bz\" (UID: \"3f85466a-6e79-4de7-b5e8-c33d02f708c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d6a18-n42bz" Oct 31 19:31:38.213010 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:38.212871 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f85466a-6e79-4de7-b5e8-c33d02f708c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-221d6a18-n42bz\" (UID: \"3f85466a-6e79-4de7-b5e8-c33d02f708c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d6a18-n42bz" Oct 31 19:31:38.213879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:38.213849 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f85466a-6e79-4de7-b5e8-c33d02f708c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-221d6a18-n42bz\" (UID: \"3f85466a-6e79-4de7-b5e8-c33d02f708c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d6a18-n42bz" Oct 31 19:31:38.215542 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:38.215520 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f85466a-6e79-4de7-b5e8-c33d02f708c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-221d6a18-n42bz\" (UID: \"3f85466a-6e79-4de7-b5e8-c33d02f708c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d6a18-n42bz" Oct 31 19:31:38.226884 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:38.226854 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8drs\" (UniqueName: \"kubernetes.io/projected/3f85466a-6e79-4de7-b5e8-c33d02f708c4-kube-api-access-w8drs\") pod \"maas-api-revoke-sub-221d6a18-n42bz\" (UID: \"3f85466a-6e79-4de7-b5e8-c33d02f708c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d6a18-n42bz" Oct 31 19:31:38.353317 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:38.353275 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d6a18-n42bz" Oct 31 19:31:38.634102 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:38.634077 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-221d6a18-n42bz"] Oct 31 19:31:38.636994 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:31:38.636961 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f85466a_6e79_4de7_b5e8_c33d02f708c4.slice/crio-fe86fc849ad8ac9ce8ab857a44238c7beac80f42952b9eb893592c148c6af0de WatchSource:0}: Error finding container fe86fc849ad8ac9ce8ab857a44238c7beac80f42952b9eb893592c148c6af0de: Status 404 returned error can't find the container with id fe86fc849ad8ac9ce8ab857a44238c7beac80f42952b9eb893592c148c6af0de Oct 31 19:31:39.188999 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:39.188887 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d6a18-n42bz" event={"ID":"3f85466a-6e79-4de7-b5e8-c33d02f708c4","Type":"ContainerStarted","Data":"13e7d2335909412834515f84beb0496854bbfbddd530460a72b536006f6ccab6"} Oct 31 19:31:39.189447 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:39.188997 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d6a18-n42bz" event={"ID":"3f85466a-6e79-4de7-b5e8-c33d02f708c4","Type":"ContainerStarted","Data":"fe86fc849ad8ac9ce8ab857a44238c7beac80f42952b9eb893592c148c6af0de"} Oct 31 19:31:39.214435 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:39.214382 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d6a18-n42bz" podStartSLOduration=1.214367108 podStartE2EDuration="1.214367108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:39.21181735 +0000 UTC m=+733.574637722" watchObservedRunningTime="2026-10-31 19:31:39.214367108 +0000 UTC m=+733.577187480" Oct 31 19:31:40.196738 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:40.196702 4910 generic.go:356] "Generic (PLEG): container finished" podID="3f85466a-6e79-4de7-b5e8-c33d02f708c4" containerID="13e7d2335909412834515f84beb0496854bbfbddd530460a72b536006f6ccab6" exitCode=0 Oct 31 19:31:40.197197 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:40.196786 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d6a18-n42bz" event={"ID":"3f85466a-6e79-4de7-b5e8-c33d02f708c4","Type":"ContainerDied","Data":"13e7d2335909412834515f84beb0496854bbfbddd530460a72b536006f6ccab6"} Oct 31 19:31:41.008351 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.008318 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8"] Oct 31 19:31:41.012361 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.012335 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:31:41.025684 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.025176 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-aeb292\"/\"distinct-w2-aeb292-kserve-self-signed-certs\"" Oct 31 19:31:41.034057 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.031826 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8"] Oct 31 19:31:41.158487 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.158451 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-tmp-dir\") pod \"distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8\" (UID: \"3803d62c-0b0f-49ce-a755-0db6506489d4\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:31:41.158874 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.158840 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-home\") pod \"distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8\" (UID: \"3803d62c-0b0f-49ce-a755-0db6506489d4\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:31:41.159167 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.159148 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-dshm\") pod \"distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8\" (UID: \"3803d62c-0b0f-49ce-a755-0db6506489d4\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:31:41.159314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.159298 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9jfn\" (UniqueName: \"kubernetes.io/projected/3803d62c-0b0f-49ce-a755-0db6506489d4-kube-api-access-l9jfn\") pod \"distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8\" (UID: \"3803d62c-0b0f-49ce-a755-0db6506489d4\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:31:41.159440 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.159426 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3803d62c-0b0f-49ce-a755-0db6506489d4-tls-certs\") pod \"distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8\" (UID: \"3803d62c-0b0f-49ce-a755-0db6506489d4\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:31:41.159567 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.159552 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-model-cache\") pod \"distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8\" (UID: \"3803d62c-0b0f-49ce-a755-0db6506489d4\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:31:41.260783 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.260694 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-tmp-dir\") pod \"distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8\" (UID: \"3803d62c-0b0f-49ce-a755-0db6506489d4\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:31:41.260783 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.260771 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-home\") pod \"distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8\" (UID: \"3803d62c-0b0f-49ce-a755-0db6506489d4\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:31:41.261372 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.260903 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-dshm\") pod \"distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8\" (UID: \"3803d62c-0b0f-49ce-a755-0db6506489d4\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:31:41.261372 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.260957 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9jfn\" (UniqueName: \"kubernetes.io/projected/3803d62c-0b0f-49ce-a755-0db6506489d4-kube-api-access-l9jfn\") pod \"distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8\" (UID: \"3803d62c-0b0f-49ce-a755-0db6506489d4\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:31:41.261372 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.260986 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3803d62c-0b0f-49ce-a755-0db6506489d4-tls-certs\") pod \"distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8\" (UID: \"3803d62c-0b0f-49ce-a755-0db6506489d4\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:31:41.261372 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.261010 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-model-cache\") pod \"distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8\" (UID: \"3803d62c-0b0f-49ce-a755-0db6506489d4\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:31:41.261577 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.261507 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-model-cache\") pod \"distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8\" (UID: \"3803d62c-0b0f-49ce-a755-0db6506489d4\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:31:41.261799 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.261765 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-tmp-dir\") pod \"distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8\" (UID: \"3803d62c-0b0f-49ce-a755-0db6506489d4\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:31:41.262113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.262092 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-home\") pod \"distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8\" (UID: \"3803d62c-0b0f-49ce-a755-0db6506489d4\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:31:41.265415 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.265387 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-dshm\") pod \"distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8\" (UID: \"3803d62c-0b0f-49ce-a755-0db6506489d4\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:31:41.270830 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.270802 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3803d62c-0b0f-49ce-a755-0db6506489d4-tls-certs\") pod \"distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8\" (UID: \"3803d62c-0b0f-49ce-a755-0db6506489d4\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:31:41.275309 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.275273 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9jfn\" (UniqueName: \"kubernetes.io/projected/3803d62c-0b0f-49ce-a755-0db6506489d4-kube-api-access-l9jfn\") pod \"distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8\" (UID: \"3803d62c-0b0f-49ce-a755-0db6506489d4\") " pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:31:41.330936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.325891 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:31:41.427419 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.427390 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d6a18-n42bz" Oct 31 19:31:41.572108 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.572075 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f85466a-6e79-4de7-b5e8-c33d02f708c4-kube-api-access-w8drs\" (UniqueName: \"kubernetes.io/projected/3f85466a-6e79-4de7-b5e8-c33d02f708c4-kube-api-access-w8drs\") pod \"3f85466a-6e79-4de7-b5e8-c33d02f708c4\" (UID: \"3f85466a-6e79-4de7-b5e8-c33d02f708c4\") " Oct 31 19:31:41.572301 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.572201 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f85466a-6e79-4de7-b5e8-c33d02f708c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f85466a-6e79-4de7-b5e8-c33d02f708c4-maas-api-openshift-ca\") pod \"3f85466a-6e79-4de7-b5e8-c33d02f708c4\" (UID: \"3f85466a-6e79-4de7-b5e8-c33d02f708c4\") " Oct 31 19:31:41.572369 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.572345 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f85466a-6e79-4de7-b5e8-c33d02f708c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f85466a-6e79-4de7-b5e8-c33d02f708c4-maas-api-xks-ca\") pod \"3f85466a-6e79-4de7-b5e8-c33d02f708c4\" (UID: \"3f85466a-6e79-4de7-b5e8-c33d02f708c4\") " Oct 31 19:31:41.572604 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.572578 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f85466a-6e79-4de7-b5e8-c33d02f708c4-maas-api-openshift-ca" pod "3f85466a-6e79-4de7-b5e8-c33d02f708c4" (UID: "3f85466a-6e79-4de7-b5e8-c33d02f708c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:31:41.572802 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.572783 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f85466a-6e79-4de7-b5e8-c33d02f708c4-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:41.574362 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.574335 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f85466a-6e79-4de7-b5e8-c33d02f708c4-maas-api-xks-ca" pod "3f85466a-6e79-4de7-b5e8-c33d02f708c4" (UID: "3f85466a-6e79-4de7-b5e8-c33d02f708c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:31:41.574689 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.574665 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f85466a-6e79-4de7-b5e8-c33d02f708c4-kube-api-access-w8drs" pod "3f85466a-6e79-4de7-b5e8-c33d02f708c4" (UID: "3f85466a-6e79-4de7-b5e8-c33d02f708c4"). InnerVolumeSpecName "kube-api-access-w8drs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:31:41.632057 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:31:41.631748 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3803d62c_0b0f_49ce_a755_0db6506489d4.slice/crio-c2ecc0ee2b5b33313c1a0580245722c55e52058f75781fb1e9affe14d0204544 WatchSource:0}: Error finding container c2ecc0ee2b5b33313c1a0580245722c55e52058f75781fb1e9affe14d0204544: Status 404 returned error can't find the container with id c2ecc0ee2b5b33313c1a0580245722c55e52058f75781fb1e9affe14d0204544 Oct 31 19:31:41.633572 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.633470 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8"] Oct 31 19:31:41.673902 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.673877 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f85466a-6e79-4de7-b5e8-c33d02f708c4-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:41.674037 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:41.673902 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8drs\" (UniqueName: \"kubernetes.io/projected/3f85466a-6e79-4de7-b5e8-c33d02f708c4-kube-api-access-w8drs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:42.237561 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:42.237516 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" event={"ID":"3803d62c-0b0f-49ce-a755-0db6506489d4","Type":"ContainerStarted","Data":"4bbdbd8807904a3d2b84a0c7644ee280c605a5794e74010f046eaeba69c58757"} Oct 31 19:31:42.237561 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:42.237563 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" event={"ID":"3803d62c-0b0f-49ce-a755-0db6506489d4","Type":"ContainerStarted","Data":"c2ecc0ee2b5b33313c1a0580245722c55e52058f75781fb1e9affe14d0204544"} Oct 31 19:31:42.239234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:42.239214 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d6a18-n42bz" Oct 31 19:31:42.239403 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:42.239212 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-221d6a18-n42bz" event={"ID":"3f85466a-6e79-4de7-b5e8-c33d02f708c4","Type":"ContainerDied","Data":"fe86fc849ad8ac9ce8ab857a44238c7beac80f42952b9eb893592c148c6af0de"} Oct 31 19:31:42.239403 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:42.239339 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe86fc849ad8ac9ce8ab857a44238c7beac80f42952b9eb893592c148c6af0de" Oct 31 19:31:42.266690 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:42.266634 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podStartSLOduration=2.266621372 podStartE2EDuration="2.266621372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:42.264123425 +0000 UTC m=+736.626943807" watchObservedRunningTime="2026-10-31 19:31:42.266621372 +0000 UTC m=+736.629441743" Oct 31 19:31:42.326466 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:42.326428 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:31:42.712233 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:31:42.712196 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:31:44.778937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:44.776993 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:31:47.954657 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:47.954614 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n"] Oct 31 19:31:47.955390 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:47.955367 4910 cpu_manager.go:414] "containerMap: removing container" podUID="3f85466a-6e79-4de7-b5e8-c33d02f708c4" containerName="revoke-keys" Oct 31 19:31:47.955479 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:47.955392 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f85466a-6e79-4de7-b5e8-c33d02f708c4" containerName="revoke-keys" Oct 31 19:31:47.955572 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:47.955559 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f85466a-6e79-4de7-b5e8-c33d02f708c4" containerName="revoke-keys" Oct 31 19:31:47.960342 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:47.960318 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n"] Oct 31 19:31:47.960472 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:47.960447 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.077104 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.077065 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk"] Oct 31 19:31:48.082006 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.081975 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.093124 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.093082 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.093329 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.093311 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.093408 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.093374 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljz9q\" (UniqueName: \"kubernetes.io/projected/32a18536-b3f0-42cc-a148-cc5fc0c24dca-kube-api-access-ljz9q\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.093509 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.093482 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.093620 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.093600 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.093672 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.093656 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.093761 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.093746 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.093801 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.093788 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.093836 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.093817 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.097845 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.097821 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk"] Oct 31 19:31:48.194988 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.194946 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.194988 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.194983 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.195207 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.195013 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.195207 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.195032 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.195207 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.195048 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.195207 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.195156 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.195422 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.195212 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.195422 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.195242 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.195422 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.195271 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce5603fa-39d7-4705-a0e3-85297ad53a78-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.195422 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.195298 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.195422 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.195381 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.195656 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.195428 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.195656 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.195454 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.195656 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.195506 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.195656 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.195541 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljz9q\" (UniqueName: \"kubernetes.io/projected/32a18536-b3f0-42cc-a148-cc5fc0c24dca-kube-api-access-ljz9q\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.195656 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.195578 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.195656 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.195612 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.195656 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.195646 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.196017 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.195683 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-797c6\" (UniqueName: \"kubernetes.io/projected/ce5603fa-39d7-4705-a0e3-85297ad53a78-kube-api-access-797c6\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.196017 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.195739 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.196017 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.195769 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.196017 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.195973 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.196231 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.196067 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.196772 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.196750 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06f99b64-h28f7"] Oct 31 19:31:48.197813 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.197787 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.197955 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.197936 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.201110 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.201090 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f99b64-h28f7" Oct 31 19:31:48.205593 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.205537 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:31:48.205969 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.205949 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:31:48.208139 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.208118 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.208796 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.208769 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljz9q\" (UniqueName: \"kubernetes.io/projected/32a18536-b3f0-42cc-a148-cc5fc0c24dca-kube-api-access-ljz9q\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.214597 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.214574 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06f99b64-h28f7"] Oct 31 19:31:48.277438 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.277410 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:48.297059 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.297033 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62ds9\" (UniqueName: \"kubernetes.io/projected/2986db8e-72fe-466c-a20a-3ca1a02c0975-kube-api-access-62ds9\") pod \"maas-api-revoke-sub-06f99b64-h28f7\" (UID: \"2986db8e-72fe-466c-a20a-3ca1a02c0975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f99b64-h28f7" Oct 31 19:31:48.297166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.297089 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2986db8e-72fe-466c-a20a-3ca1a02c0975-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06f99b64-h28f7\" (UID: \"2986db8e-72fe-466c-a20a-3ca1a02c0975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f99b64-h28f7" Oct 31 19:31:48.297266 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.297196 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce5603fa-39d7-4705-a0e3-85297ad53a78-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.297266 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.297240 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.297418 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.297298 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.297418 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.297337 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2986db8e-72fe-466c-a20a-3ca1a02c0975-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06f99b64-h28f7\" (UID: \"2986db8e-72fe-466c-a20a-3ca1a02c0975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f99b64-h28f7" Oct 31 19:31:48.297418 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.297387 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.297585 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.297420 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.297585 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.297467 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-797c6\" (UniqueName: \"kubernetes.io/projected/ce5603fa-39d7-4705-a0e3-85297ad53a78-kube-api-access-797c6\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.297585 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.297564 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.297732 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.297612 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.297732 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.297640 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.298818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.298795 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.299728 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.299702 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce5603fa-39d7-4705-a0e3-85297ad53a78-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.300035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.300009 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.300210 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.300187 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.300439 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.300386 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.300570 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.300472 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.300996 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.300979 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.312020 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.311982 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.312592 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.312572 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-797c6\" (UniqueName: \"kubernetes.io/projected/ce5603fa-39d7-4705-a0e3-85297ad53a78-kube-api-access-797c6\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.398421 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.398385 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2986db8e-72fe-466c-a20a-3ca1a02c0975-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06f99b64-h28f7\" (UID: \"2986db8e-72fe-466c-a20a-3ca1a02c0975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f99b64-h28f7" Oct 31 19:31:48.398602 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.398584 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62ds9\" (UniqueName: \"kubernetes.io/projected/2986db8e-72fe-466c-a20a-3ca1a02c0975-kube-api-access-62ds9\") pod \"maas-api-revoke-sub-06f99b64-h28f7\" (UID: \"2986db8e-72fe-466c-a20a-3ca1a02c0975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f99b64-h28f7" Oct 31 19:31:48.398649 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.398635 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2986db8e-72fe-466c-a20a-3ca1a02c0975-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06f99b64-h28f7\" (UID: \"2986db8e-72fe-466c-a20a-3ca1a02c0975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f99b64-h28f7" Oct 31 19:31:48.399789 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.399508 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2986db8e-72fe-466c-a20a-3ca1a02c0975-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06f99b64-h28f7\" (UID: \"2986db8e-72fe-466c-a20a-3ca1a02c0975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f99b64-h28f7" Oct 31 19:31:48.401827 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.401808 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2986db8e-72fe-466c-a20a-3ca1a02c0975-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06f99b64-h28f7\" (UID: \"2986db8e-72fe-466c-a20a-3ca1a02c0975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f99b64-h28f7" Oct 31 19:31:48.402001 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.401988 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:48.412066 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.412041 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62ds9\" (UniqueName: \"kubernetes.io/projected/2986db8e-72fe-466c-a20a-3ca1a02c0975-kube-api-access-62ds9\") pod \"maas-api-revoke-sub-06f99b64-h28f7\" (UID: \"2986db8e-72fe-466c-a20a-3ca1a02c0975\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f99b64-h28f7" Oct 31 19:31:48.567371 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.567344 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f99b64-h28f7" Oct 31 19:31:48.568005 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.567981 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n"] Oct 31 19:31:48.569867 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.569837 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:31:48.569978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.569895 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:31:48.569978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.569963 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:31:48.733162 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.733132 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk"] Oct 31 19:31:48.745163 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.744655 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:31:48.745163 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.744741 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:31:48.745163 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.744786 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:31:48.862255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:48.862209 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06f99b64-h28f7"] Oct 31 19:31:49.373185 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:49.373087 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f99b64-h28f7" event={"ID":"2986db8e-72fe-466c-a20a-3ca1a02c0975","Type":"ContainerStarted","Data":"400272b45d4e6674d8b50b140705ecfd8ea0ca6d717017884b37985896716b72"} Oct 31 19:31:49.373185 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:49.373149 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f99b64-h28f7" event={"ID":"2986db8e-72fe-466c-a20a-3ca1a02c0975","Type":"ContainerStarted","Data":"981557c594695edb55eb86872dfdf8e46e3ff1d602fa0889bfeb463354cda1f9"} Oct 31 19:31:49.374868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:49.374833 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" event={"ID":"ce5603fa-39d7-4705-a0e3-85297ad53a78","Type":"ContainerStarted","Data":"a28a4b487bc10e6a168d3b4871bfc85bd6870d6d79658c96328a29b279fb44a1"} Oct 31 19:31:49.375040 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:49.374892 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" event={"ID":"ce5603fa-39d7-4705-a0e3-85297ad53a78","Type":"ContainerStarted","Data":"5c8da1c650868ed4932c9800cf4d5b73607befe2f9f6f03178cdabbd89cd010f"} Oct 31 19:31:49.376859 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:49.376814 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" event={"ID":"32a18536-b3f0-42cc-a148-cc5fc0c24dca","Type":"ContainerStarted","Data":"e5ed391d101671489df9e96e4e053de9eb6110506a483aec9bfeec4f2b97a097"} Oct 31 19:31:49.376980 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:49.376874 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" event={"ID":"32a18536-b3f0-42cc-a148-cc5fc0c24dca","Type":"ContainerStarted","Data":"4c0fd0c511b8aa79c48bdc029c8cd434cda1be13cdbd8aae3ace2b706356b1fb"} Oct 31 19:31:49.395513 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:49.395451 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f99b64-h28f7" podStartSLOduration=1.395433275 podStartE2EDuration="1.395433275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:49.390807923 +0000 UTC m=+743.753628295" watchObservedRunningTime="2026-10-31 19:31:49.395433275 +0000 UTC m=+743.758253648" Oct 31 19:31:49.402739 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:49.402707 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:49.404555 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:49.404520 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 19:31:49.404684 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:49.404581 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" podUID="ce5603fa-39d7-4705-a0e3-85297ad53a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 19:31:49.514992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:49.514772 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" podStartSLOduration=2.514741355 podStartE2EDuration="2.514741355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:49.415806182 +0000 UTC m=+743.778626552" watchObservedRunningTime="2026-10-31 19:31:49.514741355 +0000 UTC m=+743.877561729" Oct 31 19:31:49.541931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:49.541829 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" podStartSLOduration=1.541806535 podStartE2EDuration="1.541806535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:49.474905545 +0000 UTC m=+743.837725920" watchObservedRunningTime="2026-10-31 19:31:49.541806535 +0000 UTC m=+743.904626909" Oct 31 19:31:50.283108 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:50.282316 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:50.285034 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:50.285002 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 19:31:50.285155 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:50.285070 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" podUID="32a18536-b3f0-42cc-a148-cc5fc0c24dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 19:31:50.393135 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:50.392802 4910 generic.go:356] "Generic (PLEG): container finished" podID="2986db8e-72fe-466c-a20a-3ca1a02c0975" containerID="400272b45d4e6674d8b50b140705ecfd8ea0ca6d717017884b37985896716b72" exitCode=0 Oct 31 19:31:50.393135 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:50.392890 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f99b64-h28f7" event={"ID":"2986db8e-72fe-466c-a20a-3ca1a02c0975","Type":"ContainerDied","Data":"400272b45d4e6674d8b50b140705ecfd8ea0ca6d717017884b37985896716b72"} Oct 31 19:31:50.403307 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:50.403279 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 19:31:50.403478 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:50.403332 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" podUID="ce5603fa-39d7-4705-a0e3-85297ad53a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 19:31:51.283137 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:51.283033 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 19:31:51.283137 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:51.283094 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" podUID="32a18536-b3f0-42cc-a148-cc5fc0c24dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 19:31:51.336067 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:51.336026 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:31:51.396555 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:51.396528 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:31:51.402313 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:51.402286 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 19:31:51.402401 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:51.402348 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" podUID="ce5603fa-39d7-4705-a0e3-85297ad53a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 19:31:51.414369 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:51.414346 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:31:51.854478 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:51.854403 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f99b64-h28f7" Oct 31 19:31:51.904732 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:51.904693 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2986db8e-72fe-466c-a20a-3ca1a02c0975-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2986db8e-72fe-466c-a20a-3ca1a02c0975-maas-api-xks-ca\") pod \"2986db8e-72fe-466c-a20a-3ca1a02c0975\" (UID: \"2986db8e-72fe-466c-a20a-3ca1a02c0975\") " Oct 31 19:31:51.905112 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:51.905085 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2986db8e-72fe-466c-a20a-3ca1a02c0975-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2986db8e-72fe-466c-a20a-3ca1a02c0975-maas-api-openshift-ca\") pod \"2986db8e-72fe-466c-a20a-3ca1a02c0975\" (UID: \"2986db8e-72fe-466c-a20a-3ca1a02c0975\") " Oct 31 19:31:51.912411 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:51.912387 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2986db8e-72fe-466c-a20a-3ca1a02c0975-kube-api-access-62ds9\" (UniqueName: \"kubernetes.io/projected/2986db8e-72fe-466c-a20a-3ca1a02c0975-kube-api-access-62ds9\") pod \"2986db8e-72fe-466c-a20a-3ca1a02c0975\" (UID: \"2986db8e-72fe-466c-a20a-3ca1a02c0975\") " Oct 31 19:31:51.917342 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:51.917214 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2986db8e-72fe-466c-a20a-3ca1a02c0975-kube-api-access-62ds9" pod "2986db8e-72fe-466c-a20a-3ca1a02c0975" (UID: "2986db8e-72fe-466c-a20a-3ca1a02c0975"). InnerVolumeSpecName "kube-api-access-62ds9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:31:51.925944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:51.922044 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2986db8e-72fe-466c-a20a-3ca1a02c0975-maas-api-xks-ca" pod "2986db8e-72fe-466c-a20a-3ca1a02c0975" (UID: "2986db8e-72fe-466c-a20a-3ca1a02c0975"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:31:51.933107 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:51.931103 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2986db8e-72fe-466c-a20a-3ca1a02c0975-maas-api-openshift-ca" pod "2986db8e-72fe-466c-a20a-3ca1a02c0975" (UID: "2986db8e-72fe-466c-a20a-3ca1a02c0975"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:31:52.014665 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:52.014630 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2986db8e-72fe-466c-a20a-3ca1a02c0975-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:52.014838 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:52.014673 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2986db8e-72fe-466c-a20a-3ca1a02c0975-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:52.014838 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:52.014689 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62ds9\" (UniqueName: \"kubernetes.io/projected/2986db8e-72fe-466c-a20a-3ca1a02c0975-kube-api-access-62ds9\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:31:52.285342 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:52.281523 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 19:31:52.285342 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:52.281586 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" podUID="32a18536-b3f0-42cc-a148-cc5fc0c24dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 19:31:52.402482 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:52.402442 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 19:31:52.402874 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:52.402511 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" podUID="ce5603fa-39d7-4705-a0e3-85297ad53a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 19:31:52.410551 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:52.410520 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f99b64-h28f7" event={"ID":"2986db8e-72fe-466c-a20a-3ca1a02c0975","Type":"ContainerDied","Data":"981557c594695edb55eb86872dfdf8e46e3ff1d602fa0889bfeb463354cda1f9"} Oct 31 19:31:52.410667 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:52.410577 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="981557c594695edb55eb86872dfdf8e46e3ff1d602fa0889bfeb463354cda1f9" Oct 31 19:31:52.410667 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:52.410526 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06f99b64-h28f7" Oct 31 19:31:53.278433 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:53.278257 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 19:31:53.278433 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:53.278318 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" podUID="32a18536-b3f0-42cc-a148-cc5fc0c24dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 19:31:53.404739 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:53.404701 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 19:31:53.405351 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:53.405299 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" podUID="ce5603fa-39d7-4705-a0e3-85297ad53a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 19:31:54.279347 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:54.279299 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 19:31:54.279647 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:54.279615 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" podUID="32a18536-b3f0-42cc-a148-cc5fc0c24dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 19:31:54.403025 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:54.402972 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 19:31:54.403810 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:54.403274 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" podUID="ce5603fa-39d7-4705-a0e3-85297ad53a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 19:31:54.822939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:54.822110 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:31:54.958106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:54.958048 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:31:55.280388 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:55.280292 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 19:31:55.280672 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:55.280635 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" podUID="32a18536-b3f0-42cc-a148-cc5fc0c24dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 19:31:55.403931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:55.403293 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 19:31:55.403931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:55.403359 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" podUID="ce5603fa-39d7-4705-a0e3-85297ad53a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 19:31:55.462969 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:55.462008 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:31:56.279106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:56.279069 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 19:31:56.279544 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:56.279166 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" podUID="32a18536-b3f0-42cc-a148-cc5fc0c24dca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 19:31:56.402832 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:56.402796 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 31 19:31:56.403117 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:56.403091 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" podUID="ce5603fa-39d7-4705-a0e3-85297ad53a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 31 19:31:57.157056 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.157015 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8"] Oct 31 19:31:57.157781 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.157764 4910 cpu_manager.go:414] "containerMap: removing container" podUID="2986db8e-72fe-466c-a20a-3ca1a02c0975" containerName="revoke-keys" Oct 31 19:31:57.157841 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.157784 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2986db8e-72fe-466c-a20a-3ca1a02c0975" containerName="revoke-keys" Oct 31 19:31:57.158024 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.158010 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="2986db8e-72fe-466c-a20a-3ca1a02c0975" containerName="revoke-keys" Oct 31 19:31:57.167110 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.167080 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:31:57.177226 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.177202 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-8730d5\"/\"distinct-2-w6-8730d5-kserve-self-signed-certs\"" Oct 31 19:31:57.181259 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.181234 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8"] Oct 31 19:31:57.283325 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.283290 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:57.283740 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.283710 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:57.284554 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.284532 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:31:57.295943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.295902 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-model-cache\") pod \"distinct-2-w6-8730d5-kserve-f8ff54685-cswm8\" (UID: \"77506413-2985-4c17-b98c-dc414769ec5d\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:31:57.296159 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.296129 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-tmp-dir\") pod \"distinct-2-w6-8730d5-kserve-f8ff54685-cswm8\" (UID: \"77506413-2985-4c17-b98c-dc414769ec5d\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:31:57.296207 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.296184 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk5wh\" (UniqueName: \"kubernetes.io/projected/77506413-2985-4c17-b98c-dc414769ec5d-kube-api-access-wk5wh\") pod \"distinct-2-w6-8730d5-kserve-f8ff54685-cswm8\" (UID: \"77506413-2985-4c17-b98c-dc414769ec5d\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:31:57.296242 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.296222 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-dshm\") pod \"distinct-2-w6-8730d5-kserve-f8ff54685-cswm8\" (UID: \"77506413-2985-4c17-b98c-dc414769ec5d\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:31:57.296282 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.296250 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-home\") pod \"distinct-2-w6-8730d5-kserve-f8ff54685-cswm8\" (UID: \"77506413-2985-4c17-b98c-dc414769ec5d\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:31:57.296319 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.296286 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77506413-2985-4c17-b98c-dc414769ec5d-tls-certs\") pod \"distinct-2-w6-8730d5-kserve-f8ff54685-cswm8\" (UID: \"77506413-2985-4c17-b98c-dc414769ec5d\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:31:57.405276 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.405233 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-tmp-dir\") pod \"distinct-2-w6-8730d5-kserve-f8ff54685-cswm8\" (UID: \"77506413-2985-4c17-b98c-dc414769ec5d\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:31:57.405469 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.405297 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk5wh\" (UniqueName: \"kubernetes.io/projected/77506413-2985-4c17-b98c-dc414769ec5d-kube-api-access-wk5wh\") pod \"distinct-2-w6-8730d5-kserve-f8ff54685-cswm8\" (UID: \"77506413-2985-4c17-b98c-dc414769ec5d\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:31:57.405469 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.405337 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-dshm\") pod \"distinct-2-w6-8730d5-kserve-f8ff54685-cswm8\" (UID: \"77506413-2985-4c17-b98c-dc414769ec5d\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:31:57.405469 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.405371 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-home\") pod \"distinct-2-w6-8730d5-kserve-f8ff54685-cswm8\" (UID: \"77506413-2985-4c17-b98c-dc414769ec5d\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:31:57.405469 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.405408 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77506413-2985-4c17-b98c-dc414769ec5d-tls-certs\") pod \"distinct-2-w6-8730d5-kserve-f8ff54685-cswm8\" (UID: \"77506413-2985-4c17-b98c-dc414769ec5d\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:31:57.405693 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.405560 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-model-cache\") pod \"distinct-2-w6-8730d5-kserve-f8ff54685-cswm8\" (UID: \"77506413-2985-4c17-b98c-dc414769ec5d\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:31:57.409047 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.408970 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-home\") pod \"distinct-2-w6-8730d5-kserve-f8ff54685-cswm8\" (UID: \"77506413-2985-4c17-b98c-dc414769ec5d\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:31:57.409454 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.409408 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-tmp-dir\") pod \"distinct-2-w6-8730d5-kserve-f8ff54685-cswm8\" (UID: \"77506413-2985-4c17-b98c-dc414769ec5d\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:31:57.411163 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.411141 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-model-cache\") pod \"distinct-2-w6-8730d5-kserve-f8ff54685-cswm8\" (UID: \"77506413-2985-4c17-b98c-dc414769ec5d\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:31:57.449040 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.447620 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk5wh\" (UniqueName: \"kubernetes.io/projected/77506413-2985-4c17-b98c-dc414769ec5d-kube-api-access-wk5wh\") pod \"distinct-2-w6-8730d5-kserve-f8ff54685-cswm8\" (UID: \"77506413-2985-4c17-b98c-dc414769ec5d\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:31:57.453286 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.453259 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-dshm\") pod \"distinct-2-w6-8730d5-kserve-f8ff54685-cswm8\" (UID: \"77506413-2985-4c17-b98c-dc414769ec5d\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:31:57.474630 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.474572 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77506413-2985-4c17-b98c-dc414769ec5d-tls-certs\") pod \"distinct-2-w6-8730d5-kserve-f8ff54685-cswm8\" (UID: \"77506413-2985-4c17-b98c-dc414769ec5d\") " pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:31:57.485390 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.485142 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:31:57.854355 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:57.854318 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8"] Oct 31 19:31:57.854486 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:31:57.854444 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77506413_2985_4c17_b98c_dc414769ec5d.slice/crio-5cb9ad3d63e083df8d1c1769f75ad8d614cb51c1591de318e3d4f1da95d2de8d WatchSource:0}: Error finding container 5cb9ad3d63e083df8d1c1769f75ad8d614cb51c1591de318e3d4f1da95d2de8d: Status 404 returned error can't find the container with id 5cb9ad3d63e083df8d1c1769f75ad8d614cb51c1591de318e3d4f1da95d2de8d Oct 31 19:31:58.048794 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:58.048740 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:58.049298 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:58.049268 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:58.050405 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:58.050381 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 19:31:58.591882 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:58.591841 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" event={"ID":"77506413-2985-4c17-b98c-dc414769ec5d","Type":"ContainerStarted","Data":"6d3c3829dd2141679e7092f74dd2f4742aa00d1440e5d42dd81ead069e4465fe"} Oct 31 19:31:58.591882 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:58.591896 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" event={"ID":"77506413-2985-4c17-b98c-dc414769ec5d","Type":"ContainerStarted","Data":"5cb9ad3d63e083df8d1c1769f75ad8d614cb51c1591de318e3d4f1da95d2de8d"} Oct 31 19:31:58.620864 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:58.620805 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podStartSLOduration=1.620787132 podStartE2EDuration="1.620787132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:58.619426647 +0000 UTC m=+752.982247037" watchObservedRunningTime="2026-10-31 19:31:58.620787132 +0000 UTC m=+752.983607507" Oct 31 19:31:59.485859 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:31:59.485811 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:32:00.550065 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.550029 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7ca570a-z76dr"] Oct 31 19:32:00.554123 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.554103 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca570a-z76dr" Oct 31 19:32:00.560226 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.560200 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:32:00.560673 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.560647 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:32:00.571464 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.571441 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7ca570a-z76dr"] Oct 31 19:32:00.665097 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.665058 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9nz7\" (UniqueName: \"kubernetes.io/projected/467f91c0-a951-4529-83be-055231ca73a4-kube-api-access-w9nz7\") pod \"maas-api-revoke-sub-f7ca570a-z76dr\" (UID: \"467f91c0-a951-4529-83be-055231ca73a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca570a-z76dr" Oct 31 19:32:00.665304 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.665275 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/467f91c0-a951-4529-83be-055231ca73a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7ca570a-z76dr\" (UID: \"467f91c0-a951-4529-83be-055231ca73a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca570a-z76dr" Oct 31 19:32:00.665437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.665418 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/467f91c0-a951-4529-83be-055231ca73a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7ca570a-z76dr\" (UID: \"467f91c0-a951-4529-83be-055231ca73a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca570a-z76dr" Oct 31 19:32:00.692070 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.692032 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws"] Oct 31 19:32:00.697251 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.697226 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:00.702270 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.702246 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-23814d\"/\"distinct-w3-23814d-kserve-self-signed-certs\"" Oct 31 19:32:00.707431 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.707406 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws"] Oct 31 19:32:00.766660 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.766623 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/467f91c0-a951-4529-83be-055231ca73a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7ca570a-z76dr\" (UID: \"467f91c0-a951-4529-83be-055231ca73a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca570a-z76dr" Oct 31 19:32:00.766832 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.766789 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/467f91c0-a951-4529-83be-055231ca73a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7ca570a-z76dr\" (UID: \"467f91c0-a951-4529-83be-055231ca73a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca570a-z76dr" Oct 31 19:32:00.766890 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.766878 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9nz7\" (UniqueName: \"kubernetes.io/projected/467f91c0-a951-4529-83be-055231ca73a4-kube-api-access-w9nz7\") pod \"maas-api-revoke-sub-f7ca570a-z76dr\" (UID: \"467f91c0-a951-4529-83be-055231ca73a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca570a-z76dr" Oct 31 19:32:00.767679 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.767648 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/467f91c0-a951-4529-83be-055231ca73a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7ca570a-z76dr\" (UID: \"467f91c0-a951-4529-83be-055231ca73a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca570a-z76dr" Oct 31 19:32:00.770281 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.770256 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/467f91c0-a951-4529-83be-055231ca73a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7ca570a-z76dr\" (UID: \"467f91c0-a951-4529-83be-055231ca73a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca570a-z76dr" Oct 31 19:32:00.780267 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.780239 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9nz7\" (UniqueName: \"kubernetes.io/projected/467f91c0-a951-4529-83be-055231ca73a4-kube-api-access-w9nz7\") pod \"maas-api-revoke-sub-f7ca570a-z76dr\" (UID: \"467f91c0-a951-4529-83be-055231ca73a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca570a-z76dr" Oct 31 19:32:00.869097 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.868843 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bfbef1c-402f-4a21-a378-e4906ec55c4c-tls-certs\") pod \"distinct-w3-23814d-kserve-8674c8fb86-2frws\" (UID: \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:00.869097 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.869035 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-home\") pod \"distinct-w3-23814d-kserve-8674c8fb86-2frws\" (UID: \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:00.869478 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.869197 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-model-cache\") pod \"distinct-w3-23814d-kserve-8674c8fb86-2frws\" (UID: \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:00.869700 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.869680 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-tmp-dir\") pod \"distinct-w3-23814d-kserve-8674c8fb86-2frws\" (UID: \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:00.870016 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.869995 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-dshm\") pod \"distinct-w3-23814d-kserve-8674c8fb86-2frws\" (UID: \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:00.870583 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.870561 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlh9z\" (UniqueName: \"kubernetes.io/projected/1bfbef1c-402f-4a21-a378-e4906ec55c4c-kube-api-access-qlh9z\") pod \"distinct-w3-23814d-kserve-8674c8fb86-2frws\" (UID: \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:00.891932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.888126 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca570a-z76dr" Oct 31 19:32:00.972317 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.971423 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bfbef1c-402f-4a21-a378-e4906ec55c4c-tls-certs\") pod \"distinct-w3-23814d-kserve-8674c8fb86-2frws\" (UID: \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:00.972317 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.971492 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-home\") pod \"distinct-w3-23814d-kserve-8674c8fb86-2frws\" (UID: \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:00.972317 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.971531 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-model-cache\") pod \"distinct-w3-23814d-kserve-8674c8fb86-2frws\" (UID: \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:00.972317 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.971545 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-tmp-dir\") pod \"distinct-w3-23814d-kserve-8674c8fb86-2frws\" (UID: \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:00.972317 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.971601 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-dshm\") pod \"distinct-w3-23814d-kserve-8674c8fb86-2frws\" (UID: \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:00.972317 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.971646 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlh9z\" (UniqueName: \"kubernetes.io/projected/1bfbef1c-402f-4a21-a378-e4906ec55c4c-kube-api-access-qlh9z\") pod \"distinct-w3-23814d-kserve-8674c8fb86-2frws\" (UID: \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:00.972733 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.972399 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-model-cache\") pod \"distinct-w3-23814d-kserve-8674c8fb86-2frws\" (UID: \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:00.972733 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.972649 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-home\") pod \"distinct-w3-23814d-kserve-8674c8fb86-2frws\" (UID: \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:00.972903 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.972862 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-tmp-dir\") pod \"distinct-w3-23814d-kserve-8674c8fb86-2frws\" (UID: \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:00.985073 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.982285 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-dshm\") pod \"distinct-w3-23814d-kserve-8674c8fb86-2frws\" (UID: \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:00.985073 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.984406 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bfbef1c-402f-4a21-a378-e4906ec55c4c-tls-certs\") pod \"distinct-w3-23814d-kserve-8674c8fb86-2frws\" (UID: \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:00.993372 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:00.992957 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlh9z\" (UniqueName: \"kubernetes.io/projected/1bfbef1c-402f-4a21-a378-e4906ec55c4c-kube-api-access-qlh9z\") pod \"distinct-w3-23814d-kserve-8674c8fb86-2frws\" (UID: \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:01.023355 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:01.023315 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:01.213287 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:01.212398 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7ca570a-z76dr"] Oct 31 19:32:01.213287 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:32:01.212848 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467f91c0_a951_4529_83be_055231ca73a4.slice/crio-d3b0cd9b875efa52e9d5b12fe6992b6c8f156c43e9688fcf2cbe65e5e0caef02 WatchSource:0}: Error finding container d3b0cd9b875efa52e9d5b12fe6992b6c8f156c43e9688fcf2cbe65e5e0caef02: Status 404 returned error can't find the container with id d3b0cd9b875efa52e9d5b12fe6992b6c8f156c43e9688fcf2cbe65e5e0caef02 Oct 31 19:32:01.335434 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:32:01.335400 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bfbef1c_402f_4a21_a378_e4906ec55c4c.slice/crio-98c18d8441673a9414552a36a894975d8707ad4770468e314a535dee7d8a09a4 WatchSource:0}: Error finding container 98c18d8441673a9414552a36a894975d8707ad4770468e314a535dee7d8a09a4: Status 404 returned error can't find the container with id 98c18d8441673a9414552a36a894975d8707ad4770468e314a535dee7d8a09a4 Oct 31 19:32:01.335637 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:01.335609 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws"] Oct 31 19:32:01.616307 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:01.616268 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca570a-z76dr" event={"ID":"467f91c0-a951-4529-83be-055231ca73a4","Type":"ContainerStarted","Data":"36f728cceae288865378ac145363f35797c62dbdbae13d8fad74c8460c1abe97"} Oct 31 19:32:01.616780 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:01.616327 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca570a-z76dr" event={"ID":"467f91c0-a951-4529-83be-055231ca73a4","Type":"ContainerStarted","Data":"d3b0cd9b875efa52e9d5b12fe6992b6c8f156c43e9688fcf2cbe65e5e0caef02"} Oct 31 19:32:01.618231 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:01.618196 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" event={"ID":"1bfbef1c-402f-4a21-a378-e4906ec55c4c","Type":"ContainerStarted","Data":"5688f4d0ce672189a34da2747a62fe8eab44fed2b5d1f7df7618e5b9ff1270d7"} Oct 31 19:32:01.618371 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:01.618237 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" event={"ID":"1bfbef1c-402f-4a21-a378-e4906ec55c4c","Type":"ContainerStarted","Data":"98c18d8441673a9414552a36a894975d8707ad4770468e314a535dee7d8a09a4"} Oct 31 19:32:01.639629 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:01.639560 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca570a-z76dr" podStartSLOduration=1.639545107 podStartE2EDuration="1.639545107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:01.63662278 +0000 UTC m=+755.999443152" watchObservedRunningTime="2026-10-31 19:32:01.639545107 +0000 UTC m=+756.002365478" Oct 31 19:32:01.659306 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:01.659248 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podStartSLOduration=1.65923142 podStartE2EDuration="1.65923142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:01.657107066 +0000 UTC m=+756.019927451" watchObservedRunningTime="2026-10-31 19:32:01.65923142 +0000 UTC m=+756.022051793" Oct 31 19:32:02.024330 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:02.024288 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:02.628537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:02.628500 4910 generic.go:356] "Generic (PLEG): container finished" podID="467f91c0-a951-4529-83be-055231ca73a4" containerID="36f728cceae288865378ac145363f35797c62dbdbae13d8fad74c8460c1abe97" exitCode=0 Oct 31 19:32:02.628990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:02.628580 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca570a-z76dr" event={"ID":"467f91c0-a951-4529-83be-055231ca73a4","Type":"ContainerDied","Data":"36f728cceae288865378ac145363f35797c62dbdbae13d8fad74c8460c1abe97"} Oct 31 19:32:02.821159 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:02.821115 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs"] Oct 31 19:32:02.825729 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:02.825704 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:02.849700 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:02.849669 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs"] Oct 31 19:32:02.997197 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:02.997111 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:02.997197 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:02.997163 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:02.997429 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:02.997310 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:02.997429 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:02.997352 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:02.997429 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:02.997415 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9dlr\" (UniqueName: \"kubernetes.io/projected/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-kube-api-access-d9dlr\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:02.997595 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:02.997487 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:02.997595 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:02.997546 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:02.997705 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:02.997640 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:02.997761 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:02.997731 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:03.098744 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.098708 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:03.098983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.098769 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:03.098983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.098803 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:03.098983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.098856 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:03.098983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.098888 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:03.098983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.098905 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9dlr\" (UniqueName: \"kubernetes.io/projected/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-kube-api-access-d9dlr\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:03.098983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.098986 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:03.099369 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.099086 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:03.099425 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.099412 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:03.099473 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.099430 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:03.099548 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.099199 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:03.099607 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.099220 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:03.099669 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.099603 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:03.099812 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.099792 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:03.101284 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.101264 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:03.101423 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.101407 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:03.116012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.115988 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9dlr\" (UniqueName: \"kubernetes.io/projected/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-kube-api-access-d9dlr\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:03.116149 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.116118 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:03.152127 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.152097 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:03.492020 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:32:03.491985 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12cfe0e_fab7_4a92_b62c_a18a3771bb1a.slice/crio-b78655a261267444c050882b766b4b20a7c8f14dc2f313f6aee83f9bdca0b69c WatchSource:0}: Error finding container b78655a261267444c050882b766b4b20a7c8f14dc2f313f6aee83f9bdca0b69c: Status 404 returned error can't find the container with id b78655a261267444c050882b766b4b20a7c8f14dc2f313f6aee83f9bdca0b69c Oct 31 19:32:03.492197 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.492174 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs"] Oct 31 19:32:03.494583 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.494548 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:03.494677 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.494646 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:03.494722 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.494690 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:03.521101 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.521070 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf"] Oct 31 19:32:03.526844 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.526821 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.546511 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.546483 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf"] Oct 31 19:32:03.607921 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.607878 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.608045 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.607947 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.608106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.608042 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.608106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.608075 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.608223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.608184 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dnj7\" (UniqueName: \"kubernetes.io/projected/09156e2c-03fe-4f93-af3a-4449f61f1e2c-kube-api-access-4dnj7\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.608280 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.608232 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.608336 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.608299 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.608375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.608360 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.608412 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.608397 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.644760 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.642304 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" event={"ID":"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a","Type":"ContainerStarted","Data":"bf5ad597b7cd5f39d29c2bccd0ea5cbf4971142f6438f85d22b568c54437cd66"} Oct 31 19:32:03.644760 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.642368 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" event={"ID":"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a","Type":"ContainerStarted","Data":"b78655a261267444c050882b766b4b20a7c8f14dc2f313f6aee83f9bdca0b69c"} Oct 31 19:32:03.709550 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.709510 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.709735 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.709565 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.709735 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.709639 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dnj7\" (UniqueName: \"kubernetes.io/projected/09156e2c-03fe-4f93-af3a-4449f61f1e2c-kube-api-access-4dnj7\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.709735 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.709680 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.709735 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.709708 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.709997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.709753 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.709997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.709802 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.709997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.709880 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.709997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.709975 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.711537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.710398 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.711698 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.711674 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.712662 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.712188 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.712662 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.712444 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.712662 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.712621 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.713724 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.713695 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.714942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.714765 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.748712 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.748646 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dnj7\" (UniqueName: \"kubernetes.io/projected/09156e2c-03fe-4f93-af3a-4449f61f1e2c-kube-api-access-4dnj7\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.749374 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.749314 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.836958 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.834355 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca570a-z76dr" Oct 31 19:32:03.865292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.865227 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" podStartSLOduration=1.8652068069999999 podStartE2EDuration="1.865206807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:03.682772569 +0000 UTC m=+758.045592942" watchObservedRunningTime="2026-10-31 19:32:03.865206807 +0000 UTC m=+758.228027183" Oct 31 19:32:03.874305 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.874277 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:03.921050 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.911754 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/467f91c0-a951-4529-83be-055231ca73a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/467f91c0-a951-4529-83be-055231ca73a4-maas-api-openshift-ca\") pod \"467f91c0-a951-4529-83be-055231ca73a4\" (UID: \"467f91c0-a951-4529-83be-055231ca73a4\") " Oct 31 19:32:03.921050 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.911938 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/467f91c0-a951-4529-83be-055231ca73a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/467f91c0-a951-4529-83be-055231ca73a4-maas-api-xks-ca\") pod \"467f91c0-a951-4529-83be-055231ca73a4\" (UID: \"467f91c0-a951-4529-83be-055231ca73a4\") " Oct 31 19:32:03.921050 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.912062 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467f91c0-a951-4529-83be-055231ca73a4-kube-api-access-w9nz7\" (UniqueName: \"kubernetes.io/projected/467f91c0-a951-4529-83be-055231ca73a4-kube-api-access-w9nz7\") pod \"467f91c0-a951-4529-83be-055231ca73a4\" (UID: \"467f91c0-a951-4529-83be-055231ca73a4\") " Oct 31 19:32:03.921050 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.912390 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/467f91c0-a951-4529-83be-055231ca73a4-maas-api-openshift-ca" pod "467f91c0-a951-4529-83be-055231ca73a4" (UID: "467f91c0-a951-4529-83be-055231ca73a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:32:03.921050 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.912553 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/467f91c0-a951-4529-83be-055231ca73a4-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:32:03.921050 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.914944 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467f91c0-a951-4529-83be-055231ca73a4-kube-api-access-w9nz7" pod "467f91c0-a951-4529-83be-055231ca73a4" (UID: "467f91c0-a951-4529-83be-055231ca73a4"). InnerVolumeSpecName "kube-api-access-w9nz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:32:03.921050 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:03.916503 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/467f91c0-a951-4529-83be-055231ca73a4-maas-api-xks-ca" pod "467f91c0-a951-4529-83be-055231ca73a4" (UID: "467f91c0-a951-4529-83be-055231ca73a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:32:04.013667 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:04.013633 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9nz7\" (UniqueName: \"kubernetes.io/projected/467f91c0-a951-4529-83be-055231ca73a4-kube-api-access-w9nz7\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:32:04.013667 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:04.013665 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/467f91c0-a951-4529-83be-055231ca73a4-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:32:04.153359 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:04.153315 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:04.155597 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:04.155567 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 19:32:04.155755 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:04.155627 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" podUID="b12cfe0e-fab7-4a92-b62c-a18a3771bb1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 19:32:04.199020 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:04.198978 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf"] Oct 31 19:32:04.236051 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:04.233773 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:04.236051 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:04.233839 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:04.236051 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:04.233868 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:04.665676 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:04.665583 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca570a-z76dr" Oct 31 19:32:04.665676 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:04.665622 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7ca570a-z76dr" event={"ID":"467f91c0-a951-4529-83be-055231ca73a4","Type":"ContainerDied","Data":"d3b0cd9b875efa52e9d5b12fe6992b6c8f156c43e9688fcf2cbe65e5e0caef02"} Oct 31 19:32:04.666194 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:04.665681 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3b0cd9b875efa52e9d5b12fe6992b6c8f156c43e9688fcf2cbe65e5e0caef02" Oct 31 19:32:04.668558 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:04.667877 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" event={"ID":"09156e2c-03fe-4f93-af3a-4449f61f1e2c","Type":"ContainerStarted","Data":"1ca447bd12143f53b0b0c4b345c1fe9e80305c79c26614cb1d824b359454ec3a"} Oct 31 19:32:04.668558 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:04.667943 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" event={"ID":"09156e2c-03fe-4f93-af3a-4449f61f1e2c","Type":"ContainerStarted","Data":"87e3c7b7812e781f9c8b6cfa0621950e01a8f705df4f82cc9163ecddae5b8fb5"} Oct 31 19:32:04.698566 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:04.698516 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" podStartSLOduration=1.698502511 podStartE2EDuration="1.698502511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:04.694199331 +0000 UTC m=+759.057019702" watchObservedRunningTime="2026-10-31 19:32:04.698502511 +0000 UTC m=+759.061322882" Oct 31 19:32:04.875136 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:04.875102 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:05.156932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:05.155382 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 19:32:05.156932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:05.155447 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" podUID="b12cfe0e-fab7-4a92-b62c-a18a3771bb1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 19:32:05.880129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:05.880040 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 19:32:05.880832 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:05.880787 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" podUID="09156e2c-03fe-4f93-af3a-4449f61f1e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 19:32:06.161027 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:06.154488 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 19:32:06.161027 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:06.154551 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" podUID="b12cfe0e-fab7-4a92-b62c-a18a3771bb1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 19:32:06.880931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:06.880357 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 19:32:06.880931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:06.880421 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" podUID="09156e2c-03fe-4f93-af3a-4449f61f1e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 19:32:07.156991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:07.156319 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 19:32:07.156991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:07.156387 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" podUID="b12cfe0e-fab7-4a92-b62c-a18a3771bb1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 19:32:07.487983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:07.487015 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:32:07.500973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:07.500946 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:32:07.875350 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:07.875309 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 19:32:07.875676 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:07.875635 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" podUID="09156e2c-03fe-4f93-af3a-4449f61f1e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 19:32:07.961941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:07.961277 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:32:08.167994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:08.165130 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 19:32:08.167994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:08.165191 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" podUID="b12cfe0e-fab7-4a92-b62c-a18a3771bb1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 19:32:08.886760 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:08.886310 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 19:32:08.886760 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:08.886376 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" podUID="09156e2c-03fe-4f93-af3a-4449f61f1e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 19:32:09.156424 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:09.156329 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 19:32:09.157031 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:09.156996 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" podUID="b12cfe0e-fab7-4a92-b62c-a18a3771bb1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 19:32:09.791941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:09.789575 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:32:09.791941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:09.790963 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:09.795766 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:09.795438 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:09.825948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:09.816460 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:09.875393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:09.875352 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 19:32:09.878409 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:09.878370 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" podUID="09156e2c-03fe-4f93-af3a-4449f61f1e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 19:32:09.921421 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:09.921377 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:10.169934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:10.167450 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 19:32:10.169934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:10.167515 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" podUID="b12cfe0e-fab7-4a92-b62c-a18a3771bb1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 19:32:10.464994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:10.461174 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:10.494940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:10.493012 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:10.877431 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:10.877395 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 19:32:10.877772 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:10.877739 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" podUID="09156e2c-03fe-4f93-af3a-4449f61f1e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 19:32:11.026022 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:11.025988 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:11.085736 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:11.085705 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:11.172006 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:11.169295 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 19:32:11.172006 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:11.169356 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" podUID="b12cfe0e-fab7-4a92-b62c-a18a3771bb1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 19:32:11.875479 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:11.875436 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 19:32:11.875725 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:11.875687 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" podUID="09156e2c-03fe-4f93-af3a-4449f61f1e2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 19:32:11.908796 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:11.908767 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:12.717064 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:32:12.717028 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:32:12.896934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:12.894856 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:12.896934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:12.896574 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:12.902933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:12.902305 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 19:32:13.012159 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:13.012070 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:13.012756 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:13.012732 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:13.015399 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:13.015377 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:32:18.219404 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.219372 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s"] Oct 31 19:32:18.219990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.219975 4910 cpu_manager.go:414] "containerMap: removing container" podUID="467f91c0-a951-4529-83be-055231ca73a4" containerName="revoke-keys" Oct 31 19:32:18.220053 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.219990 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467f91c0-a951-4529-83be-055231ca73a4" containerName="revoke-keys" Oct 31 19:32:18.220091 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.220086 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="467f91c0-a951-4529-83be-055231ca73a4" containerName="revoke-keys" Oct 31 19:32:18.224249 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.224227 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.249395 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.249331 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s"] Oct 31 19:32:18.296960 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.296895 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.297153 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.296973 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.297153 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.297122 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.297272 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.297153 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/638cecc0-b837-4dfb-9651-793b6f2986b2-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.297272 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.297229 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.297272 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.297250 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.297437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.297274 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpgtg\" (UniqueName: \"kubernetes.io/projected/638cecc0-b837-4dfb-9651-793b6f2986b2-kube-api-access-hpgtg\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.297437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.297358 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.297437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.297392 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.388605 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.388569 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx"] Oct 31 19:32:18.392862 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.392840 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.398353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.398306 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.398489 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.398427 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.398489 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.398455 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/638cecc0-b837-4dfb-9651-793b6f2986b2-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.398604 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.398556 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.398665 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.398591 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.398665 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.398636 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpgtg\" (UniqueName: \"kubernetes.io/projected/638cecc0-b837-4dfb-9651-793b6f2986b2-kube-api-access-hpgtg\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.398769 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.398666 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.398823 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.398775 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.398823 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.398784 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.398968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.398833 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.398968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.398893 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.399071 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.399056 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.399260 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.399238 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.399861 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.399841 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/638cecc0-b837-4dfb-9651-793b6f2986b2-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.402271 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.402244 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.404032 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.404008 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.409472 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.409450 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx"] Oct 31 19:32:18.419377 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.419342 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpgtg\" (UniqueName: \"kubernetes.io/projected/638cecc0-b837-4dfb-9651-793b6f2986b2-kube-api-access-hpgtg\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.419925 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.419888 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.437039 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.437014 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h"] Oct 31 19:32:18.445228 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.445205 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.468239 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.467702 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h"] Oct 31 19:32:18.500720 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.500627 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.501005 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.500973 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.501140 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.501122 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.501242 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.501228 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.501439 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.501410 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.501579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.501564 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.501701 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.501676 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqs64\" (UniqueName: \"kubernetes.io/projected/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-kube-api-access-wqs64\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.501810 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.501713 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.501810 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.501763 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.501810 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.501785 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.502003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.501837 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.502003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.501872 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.502003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.501904 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.502003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.501962 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.502136 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.502052 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.502136 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.502108 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.502212 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.502135 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.502212 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.502199 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v7b7\" (UniqueName: \"kubernetes.io/projected/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-kube-api-access-8v7b7\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.554899 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.554860 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:18.604160 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.604107 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.604309 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.604168 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqs64\" (UniqueName: \"kubernetes.io/projected/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-kube-api-access-wqs64\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.604309 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.604235 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.604691 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.604671 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.604748 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.604730 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.604806 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.604770 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.604860 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.604816 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.604932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.604853 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.604932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.604889 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.605045 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.604930 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.605045 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.605029 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.605144 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.605087 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.605144 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.605109 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.605248 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.605175 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v7b7\" (UniqueName: \"kubernetes.io/projected/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-kube-api-access-8v7b7\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.605248 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.605208 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.605348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.605286 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.605348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.605317 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.605348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.605343 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.605507 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.605381 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.605507 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.605397 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.605734 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.605716 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.605867 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.605848 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.606051 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.606027 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.606189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.606145 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.606453 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.606390 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.606745 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.606701 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.607133 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.607072 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.607832 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.607587 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.607832 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.607791 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.607996 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.607842 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.608864 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.608845 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.609079 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.609057 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.625962 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.625515 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.625962 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.625721 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v7b7\" (UniqueName: \"kubernetes.io/projected/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-kube-api-access-8v7b7\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.637702 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.637642 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.640584 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.640521 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqs64\" (UniqueName: \"kubernetes.io/projected/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-kube-api-access-wqs64\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.736241 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.736204 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr"] Oct 31 19:32:18.742139 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.742118 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.752841 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.752774 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:18.764369 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.764346 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr"] Oct 31 19:32:18.776981 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.774983 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:18.809058 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.808965 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.809195 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.809089 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.809195 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.809127 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.809195 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.809185 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.809348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.809233 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3653569-1618-48e9-803d-5cb94a356041-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.809348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.809299 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3653569-1618-48e9-803d-5cb94a356041-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.809434 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.809366 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.809434 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.809404 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3653569-1618-48e9-803d-5cb94a356041-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.809561 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.809541 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t5kd\" (UniqueName: \"kubernetes.io/projected/c3653569-1618-48e9-803d-5cb94a356041-kube-api-access-2t5kd\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.854754 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:32:18.854686 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638cecc0_b837_4dfb_9651_793b6f2986b2.slice/crio-90d737aaa42a428a0655c62579e9015cdd3743f1dcc3dae288978db6e48b6565 WatchSource:0}: Error finding container 90d737aaa42a428a0655c62579e9015cdd3743f1dcc3dae288978db6e48b6565: Status 404 returned error can't find the container with id 90d737aaa42a428a0655c62579e9015cdd3743f1dcc3dae288978db6e48b6565 Oct 31 19:32:18.859151 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.856961 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s"] Oct 31 19:32:18.864153 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.864093 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:18.864380 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.864355 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:18.864504 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.864487 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:18.911012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.910875 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t5kd\" (UniqueName: \"kubernetes.io/projected/c3653569-1618-48e9-803d-5cb94a356041-kube-api-access-2t5kd\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.914265 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.911634 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.914265 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.911708 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.914265 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.911725 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.914265 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.911752 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.914265 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.911780 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3653569-1618-48e9-803d-5cb94a356041-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.914265 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.911825 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3653569-1618-48e9-803d-5cb94a356041-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.914265 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.911856 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.914265 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.911873 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3653569-1618-48e9-803d-5cb94a356041-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.914265 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.912484 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3653569-1618-48e9-803d-5cb94a356041-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.914265 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.912710 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.914265 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.913296 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.914265 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.913400 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.914265 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.913540 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.921975 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.916232 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.928674 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.925883 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t5kd\" (UniqueName: \"kubernetes.io/projected/c3653569-1618-48e9-803d-5cb94a356041-kube-api-access-2t5kd\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.939730 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.939671 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3653569-1618-48e9-803d-5cb94a356041-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.949409 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.949362 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3653569-1618-48e9-803d-5cb94a356041-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:18.954335 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:18.954271 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" event={"ID":"638cecc0-b837-4dfb-9651-793b6f2986b2","Type":"ContainerStarted","Data":"90d737aaa42a428a0655c62579e9015cdd3743f1dcc3dae288978db6e48b6565"} Oct 31 19:32:19.058979 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:19.058239 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:19.110957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:19.110736 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx"] Oct 31 19:32:19.112306 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:19.112269 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:19.112528 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:19.112502 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:19.112638 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:19.112586 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:19.179065 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:32:19.162544 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab4bc46_8c70_4f5a_8e0d_f852190f8c16.slice/crio-034e7e8358a5fa7276efb23ae1526420b255b00985bb2e3b25d16768d67bb097 WatchSource:0}: Error finding container 034e7e8358a5fa7276efb23ae1526420b255b00985bb2e3b25d16768d67bb097: Status 404 returned error can't find the container with id 034e7e8358a5fa7276efb23ae1526420b255b00985bb2e3b25d16768d67bb097 Oct 31 19:32:19.179065 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:19.164571 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h"] Oct 31 19:32:19.179065 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:19.164987 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:19.179065 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:19.165073 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:19.179065 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:19.165118 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:19.394982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:19.394766 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr"] Oct 31 19:32:19.408286 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:19.407163 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:19.408286 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:19.407782 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:19.408286 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:19.407859 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:20.066938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:20.063217 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" event={"ID":"638cecc0-b837-4dfb-9651-793b6f2986b2","Type":"ContainerStarted","Data":"098e5ca4398fca1404631c9535d37d81fc8b88296304452244679b31867644aa"} Oct 31 19:32:20.066938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:20.066328 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" event={"ID":"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94","Type":"ContainerStarted","Data":"ad1a9a9e89e144c56863eac13e70e8b50eff089d96858192c0bc62c97d2bc165"} Oct 31 19:32:20.066938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:20.066383 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" event={"ID":"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94","Type":"ContainerStarted","Data":"408581dc132d3db05ee34d971321e4cae2657021a9cb89063ccf6cdd6f740ac9"} Oct 31 19:32:20.081936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:20.078811 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" event={"ID":"c3653569-1618-48e9-803d-5cb94a356041","Type":"ContainerStarted","Data":"4e1129c1e6cf16ad431c16d9f3a49e3559a535c1eba14676af9fe0b87196a128"} Oct 31 19:32:20.081936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:20.078874 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" event={"ID":"c3653569-1618-48e9-803d-5cb94a356041","Type":"ContainerStarted","Data":"53dc777aea019832d892256e970fc07c9b7b72ccdcbd10d8522603d6b8d202a3"} Oct 31 19:32:20.090934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:20.088102 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" event={"ID":"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16","Type":"ContainerStarted","Data":"42d858a26dd2495bb3c2775ccee03fa7d591627c25db84bd9eef77f8831fb972"} Oct 31 19:32:20.090934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:20.088169 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" event={"ID":"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16","Type":"ContainerStarted","Data":"034e7e8358a5fa7276efb23ae1526420b255b00985bb2e3b25d16768d67bb097"} Oct 31 19:32:20.100036 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:20.099973 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" podStartSLOduration=2.099951859 podStartE2EDuration="2.099951859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:20.08938618 +0000 UTC m=+774.452206553" watchObservedRunningTime="2026-10-31 19:32:20.099951859 +0000 UTC m=+774.462772233" Oct 31 19:32:20.124455 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:20.124395 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" podStartSLOduration=2.124375553 podStartE2EDuration="2.124375553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:20.122440436 +0000 UTC m=+774.485260841" watchObservedRunningTime="2026-10-31 19:32:20.124375553 +0000 UTC m=+774.487195928" Oct 31 19:32:20.184973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:20.184105 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" podStartSLOduration=2.184084753 podStartE2EDuration="2.184084753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:20.181767299 +0000 UTC m=+774.544587672" watchObservedRunningTime="2026-10-31 19:32:20.184084753 +0000 UTC m=+774.546905127" Oct 31 19:32:20.190542 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:20.190483 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" podStartSLOduration=2.190464978 podStartE2EDuration="2.190464978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:20.152107997 +0000 UTC m=+774.514928370" watchObservedRunningTime="2026-10-31 19:32:20.190464978 +0000 UTC m=+774.553285350" Oct 31 19:32:20.556995 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:20.556275 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:20.559929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:20.559032 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 19:32:20.559929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:20.559098 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" podUID="638cecc0-b837-4dfb-9651-793b6f2986b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 19:32:20.764560 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:20.762014 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:20.776938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:20.774052 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 19:32:20.776938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:20.774122 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" podUID="09020e9b-b4e8-4cbc-946e-73f3f7e8fc94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 19:32:20.776938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:20.775535 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:20.813024 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:20.806978 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 19:32:20.813024 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:20.807040 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" podUID="6ab4bc46-8c70-4f5a-8e0d-f852190f8c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 19:32:21.062527 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:21.062497 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:21.062874 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:21.062855 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 19:32:21.063020 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:21.062998 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" podUID="c3653569-1618-48e9-803d-5cb94a356041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 19:32:21.407957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:21.407725 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:32:21.565935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:21.560327 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 19:32:21.565935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:21.560396 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" podUID="638cecc0-b837-4dfb-9651-793b6f2986b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 19:32:21.756995 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:21.755016 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:21.756995 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:21.756394 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 19:32:21.756995 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:21.756443 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" podUID="09020e9b-b4e8-4cbc-946e-73f3f7e8fc94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 19:32:21.791936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:21.782097 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:21.791936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:21.782926 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:21.791936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:21.783801 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 19:32:21.791936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:21.783834 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" podUID="6ab4bc46-8c70-4f5a-8e0d-f852190f8c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 19:32:21.811936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:21.810999 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:21.893935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:21.893104 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:21.923935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:21.917992 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:21.996932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:21.996013 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:22.040392 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.040267 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:22.068934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.063965 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 19:32:22.068934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.064027 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" podUID="c3653569-1618-48e9-803d-5cb94a356041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 19:32:22.089936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.083157 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:22.345286 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.345244 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:22.411112 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.411056 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:22.412147 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.412116 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:32:22.461799 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.461758 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:22.580977 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.574343 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 19:32:22.580977 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.574421 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" podUID="638cecc0-b837-4dfb-9651-793b6f2986b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 19:32:22.783993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.774036 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 19:32:22.783993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.774106 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" podUID="09020e9b-b4e8-4cbc-946e-73f3f7e8fc94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 19:32:22.805936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.789002 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:22.805936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.795267 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 19:32:22.805936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.795332 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" podUID="6ab4bc46-8c70-4f5a-8e0d-f852190f8c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 19:32:22.805936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.795387 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:32:22.805936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.796091 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:32:22.805936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.796157 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:32:22.806472 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.806436 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:32:22.806945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.806925 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:32:22.815005 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.814975 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:22.815467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.815448 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:32:22.921934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.921285 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:22.921934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.921581 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:32:22.945936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.929617 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:22.945936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:22.938023 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:23.000937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:23.000327 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:32:23.008936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:23.006185 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:32:23.099041 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:23.098997 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:23.101226 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:23.099826 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:32:23.132291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:23.131435 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 19:32:23.132291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:23.131493 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" podUID="c3653569-1618-48e9-803d-5cb94a356041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 19:32:23.285047 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:23.285016 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:32:23.432903 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:23.432673 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:23.561091 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:23.561052 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 19:32:23.581255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:23.581209 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" podUID="638cecc0-b837-4dfb-9651-793b6f2986b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 19:32:23.765788 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:23.765681 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 19:32:23.766420 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:23.766026 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" podUID="09020e9b-b4e8-4cbc-946e-73f3f7e8fc94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 19:32:23.783819 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:23.783786 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 19:32:23.784042 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:23.784018 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" podUID="6ab4bc46-8c70-4f5a-8e0d-f852190f8c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 19:32:23.806553 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:23.806511 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:23.807418 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:23.807385 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:23.808936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:23.808893 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:23.836938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:23.826672 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:23.928062 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:23.928014 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:32:23.932039 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:23.931998 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:24.059946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:24.059406 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 19:32:24.059946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:24.059479 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" podUID="c3653569-1618-48e9-803d-5cb94a356041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 19:32:24.102937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:24.101018 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:32:24.445403 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:24.445307 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:24.558666 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:24.558621 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 19:32:24.559048 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:24.559003 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" podUID="638cecc0-b837-4dfb-9651-793b6f2986b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 19:32:24.635933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:24.632496 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:32:24.758413 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:24.758323 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 19:32:24.758662 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:24.758630 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" podUID="09020e9b-b4e8-4cbc-946e-73f3f7e8fc94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 19:32:24.798935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:24.777777 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 19:32:24.798935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:24.777827 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" podUID="6ab4bc46-8c70-4f5a-8e0d-f852190f8c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 19:32:24.835936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:24.834579 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:32:24.873932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:24.847126 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:32:24.877933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:24.874697 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:24.959933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:24.956101 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:32:25.011655 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:25.011560 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:25.076640 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:25.076603 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 19:32:25.076898 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:25.076863 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" podUID="c3653569-1618-48e9-803d-5cb94a356041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 19:32:25.112945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:25.094943 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:25.113244 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:25.102733 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:25.284187 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:25.284075 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:25.449257 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:25.448997 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:32:25.460049 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:25.460002 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:32:25.562002 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:25.561946 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 19:32:25.562184 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:25.562024 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" podUID="638cecc0-b837-4dfb-9651-793b6f2986b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 19:32:25.779371 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:25.779179 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 19:32:25.792961 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:25.784038 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" podUID="09020e9b-b4e8-4cbc-946e-73f3f7e8fc94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 19:32:25.792961 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:25.780083 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 19:32:25.792961 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:25.784441 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" podUID="6ab4bc46-8c70-4f5a-8e0d-f852190f8c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 19:32:25.846002 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:25.835019 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:25.849060 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:25.849018 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:25.885554 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:25.877595 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:25.958194 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:25.958136 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:26.022038 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:26.021991 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:26.060419 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:26.060386 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 19:32:26.060727 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:26.060683 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" podUID="c3653569-1618-48e9-803d-5cb94a356041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 19:32:26.117022 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:26.114485 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:32:26.451460 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:26.451015 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:26.469930 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:26.468997 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:26.470135 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:26.469948 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:32:26.494943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:26.494429 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:32:26.515756 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:26.513595 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:32:26.550931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:26.550886 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:32:26.576312 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:26.576282 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 19:32:26.576556 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:26.576528 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" podUID="638cecc0-b837-4dfb-9651-793b6f2986b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 19:32:26.832106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:26.832066 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 19:32:26.845031 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:26.844986 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" podUID="6ab4bc46-8c70-4f5a-8e0d-f852190f8c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 19:32:26.853027 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:26.837572 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 19:32:26.861044 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:26.860997 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" podUID="09020e9b-b4e8-4cbc-946e-73f3f7e8fc94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 19:32:26.869097 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:26.869053 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:26.878975 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:26.869517 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:32:26.975809 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:26.881886 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:32:26.979775 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:26.976972 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:27.176313 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:27.176228 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 19:32:27.285237 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:27.285201 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" podUID="c3653569-1618-48e9-803d-5cb94a356041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 19:32:27.314101 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:27.279896 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:27.468989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:27.467456 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:27.470472 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:27.469502 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:27.496436 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:27.496389 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:27.569331 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:27.569280 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 19:32:27.569662 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:27.569610 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" podUID="638cecc0-b837-4dfb-9651-793b6f2986b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 19:32:27.640139 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:27.640066 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:27.789991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:27.787358 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 19:32:27.789991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:27.787421 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" podUID="09020e9b-b4e8-4cbc-946e-73f3f7e8fc94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 19:32:27.789991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:27.788925 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 19:32:27.789991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:27.788974 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" podUID="6ab4bc46-8c70-4f5a-8e0d-f852190f8c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 19:32:27.846081 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:27.846035 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:27.875606 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:27.875567 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 19:32:27.985104 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:27.985054 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:27.986531 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:27.985998 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:32:28.077635 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.077600 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 19:32:28.077898 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.077870 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" podUID="c3653569-1618-48e9-803d-5cb94a356041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 19:32:28.274513 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.274478 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:32:28.316938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.316878 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:32:28.317781 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.317749 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:32:28.318688 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.318651 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:28.318875 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.318859 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:32:28.475007 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.469044 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:28.475007 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.470280 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:32:28.496663 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.496606 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:28.497049 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.497019 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:28.497579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.497557 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:32:28.562906 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.562872 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 19:32:28.580659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.563430 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:28.584346 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.580608 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" podUID="638cecc0-b837-4dfb-9651-793b6f2986b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 19:32:28.634740 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.634709 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:32:28.655903 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.655863 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:28.656781 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.656755 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:32:28.762143 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.762055 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 19:32:28.766189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.766136 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" podUID="09020e9b-b4e8-4cbc-946e-73f3f7e8fc94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 19:32:28.795860 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.795815 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 31 19:32:28.796160 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.796105 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" podUID="6ab4bc46-8c70-4f5a-8e0d-f852190f8c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 31 19:32:28.799733 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.799713 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:32:28.879017 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.854071 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:28.879017 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.854930 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:32:28.928300 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:28.928178 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:32:29.063990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:29.063316 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 31 19:32:29.063990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:29.063374 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" podUID="c3653569-1618-48e9-803d-5cb94a356041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 31 19:32:29.134965 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:29.120014 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:32:29.134965 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:29.120640 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:32:29.157931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:29.156077 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:32:29.568550 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:29.567186 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:29.568550 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:29.568410 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:29.570110 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:29.570091 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:32:29.778532 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:29.777897 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:29.780080 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:29.779036 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:29.780080 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:29.779711 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:32:29.786724 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:29.786578 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:29.787531 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:29.787513 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:29.788293 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:29.788273 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:32:30.111352 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:30.111320 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:30.112244 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:30.112220 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:30.113263 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:30.113234 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 19:32:32.374354 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:32.374033 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:32.640103 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:32.639987 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:32:32.820140 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:32.818999 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:32.941121 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:32.941008 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:33.122347 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:33.122304 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:32:33.463932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:33.458997 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:33.807569 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:33.805196 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:33.989019 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:33.988969 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9"] Oct 31 19:32:34.062981 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.048193 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.137932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.099797 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt"] Oct 31 19:32:34.137932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.120328 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7wh5\" (UniqueName: \"kubernetes.io/projected/012ba66a-ce6f-4261-9516-e8f0aa8ff815-kube-api-access-p7wh5\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.137932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.120432 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.137932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.120490 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.137932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.120696 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.137932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.120741 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.137932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.120804 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.137932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.120961 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.137932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.121173 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.137932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.121221 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.159246 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.159218 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.200117 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.200085 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9"] Oct 31 19:32:34.207015 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.206983 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt"] Oct 31 19:32:34.266795 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.266756 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7wh5\" (UniqueName: \"kubernetes.io/projected/012ba66a-ce6f-4261-9516-e8f0aa8ff815-kube-api-access-p7wh5\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.267108 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.267080 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.267342 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.267323 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d95c12-a91e-44a2-94a4-001d4c3ec257-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.267480 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.267463 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.267619 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.267604 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.267873 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.267851 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2sbq\" (UniqueName: \"kubernetes.io/projected/78d95c12-a91e-44a2-94a4-001d4c3ec257-kube-api-access-q2sbq\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.276012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.274540 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.276012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.274580 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.276012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.274647 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.276012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.274677 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.276012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.274725 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.276012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.274803 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.276012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.274868 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.276012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.274937 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.276012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.274981 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.276012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.275012 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.276012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.275049 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.276012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.275072 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.276012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.275549 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.276012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.275787 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.276012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.276019 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.277519 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.277487 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.277922 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.277888 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.292259 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.292221 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.292702 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.292671 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7wh5\" (UniqueName: \"kubernetes.io/projected/012ba66a-ce6f-4261-9516-e8f0aa8ff815-kube-api-access-p7wh5\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.324895 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.324804 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.349520 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.349485 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.381971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.377694 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.381971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.377762 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.381971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.377802 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.381971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.377868 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.381971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.377946 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d95c12-a91e-44a2-94a4-001d4c3ec257-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.381971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.378001 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2sbq\" (UniqueName: \"kubernetes.io/projected/78d95c12-a91e-44a2-94a4-001d4c3ec257-kube-api-access-q2sbq\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.381971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.378094 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.381971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.378136 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.381971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.378207 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.381971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.378697 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.381971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.378993 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.381971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.379596 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.383840 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.383814 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d95c12-a91e-44a2-94a4-001d4c3ec257-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.384663 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.384637 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.405546 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.399252 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.406869 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.406841 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.429655 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.429623 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2sbq\" (UniqueName: \"kubernetes.io/projected/78d95c12-a91e-44a2-94a4-001d4c3ec257-kube-api-access-q2sbq\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.445812 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.445743 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.487966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.487677 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:34.561504 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.561153 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:34.891087 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.890700 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9"] Oct 31 19:32:34.892344 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.891640 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:34.892344 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.891806 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:34.892344 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.891854 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:34.980162 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.980117 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt"] Oct 31 19:32:34.982268 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:32:34.981161 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d95c12_a91e_44a2_94a4_001d4c3ec257.slice/crio-f39ed49f90664aa9c89c0325bb0f3953bae3208a17a94cd8fdee5e50dced2c54 WatchSource:0}: Error finding container f39ed49f90664aa9c89c0325bb0f3953bae3208a17a94cd8fdee5e50dced2c54: Status 404 returned error can't find the container with id f39ed49f90664aa9c89c0325bb0f3953bae3208a17a94cd8fdee5e50dced2c54 Oct 31 19:32:34.984895 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.984861 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:34.985018 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.984957 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:34.985018 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:34.985010 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:35.163220 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.162453 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb"] Oct 31 19:32:35.171262 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.171229 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:32:35.173802 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.173773 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb"] Oct 31 19:32:35.187179 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.187152 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-23843f\"/\"distinct-w0-23843f-kserve-self-signed-certs\"" Oct 31 19:32:35.217314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.217246 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jwwh\" (UniqueName: \"kubernetes.io/projected/e7673edf-f90d-4902-a858-c8ec9b5a289b-kube-api-access-7jwwh\") pod \"distinct-w0-23843f-kserve-bb7f85f67-wl7fb\" (UID: \"e7673edf-f90d-4902-a858-c8ec9b5a289b\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:32:35.217510 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.217489 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-dshm\") pod \"distinct-w0-23843f-kserve-bb7f85f67-wl7fb\" (UID: \"e7673edf-f90d-4902-a858-c8ec9b5a289b\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:32:35.217583 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.217533 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-tmp-dir\") pod \"distinct-w0-23843f-kserve-bb7f85f67-wl7fb\" (UID: \"e7673edf-f90d-4902-a858-c8ec9b5a289b\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:32:35.217583 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.217570 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-model-cache\") pod \"distinct-w0-23843f-kserve-bb7f85f67-wl7fb\" (UID: \"e7673edf-f90d-4902-a858-c8ec9b5a289b\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:32:35.217693 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.217661 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-home\") pod \"distinct-w0-23843f-kserve-bb7f85f67-wl7fb\" (UID: \"e7673edf-f90d-4902-a858-c8ec9b5a289b\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:32:35.217693 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.217688 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7673edf-f90d-4902-a858-c8ec9b5a289b-tls-certs\") pod \"distinct-w0-23843f-kserve-bb7f85f67-wl7fb\" (UID: \"e7673edf-f90d-4902-a858-c8ec9b5a289b\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:32:35.339235 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.339199 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-dshm\") pod \"distinct-w0-23843f-kserve-bb7f85f67-wl7fb\" (UID: \"e7673edf-f90d-4902-a858-c8ec9b5a289b\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:32:35.339426 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.339263 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-tmp-dir\") pod \"distinct-w0-23843f-kserve-bb7f85f67-wl7fb\" (UID: \"e7673edf-f90d-4902-a858-c8ec9b5a289b\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:32:35.339426 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.339305 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-model-cache\") pod \"distinct-w0-23843f-kserve-bb7f85f67-wl7fb\" (UID: \"e7673edf-f90d-4902-a858-c8ec9b5a289b\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:32:35.339426 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.339420 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-home\") pod \"distinct-w0-23843f-kserve-bb7f85f67-wl7fb\" (UID: \"e7673edf-f90d-4902-a858-c8ec9b5a289b\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:32:35.339592 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.339450 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7673edf-f90d-4902-a858-c8ec9b5a289b-tls-certs\") pod \"distinct-w0-23843f-kserve-bb7f85f67-wl7fb\" (UID: \"e7673edf-f90d-4902-a858-c8ec9b5a289b\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:32:35.339592 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.339531 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jwwh\" (UniqueName: \"kubernetes.io/projected/e7673edf-f90d-4902-a858-c8ec9b5a289b-kube-api-access-7jwwh\") pod \"distinct-w0-23843f-kserve-bb7f85f67-wl7fb\" (UID: \"e7673edf-f90d-4902-a858-c8ec9b5a289b\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:32:35.342933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.340556 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-tmp-dir\") pod \"distinct-w0-23843f-kserve-bb7f85f67-wl7fb\" (UID: \"e7673edf-f90d-4902-a858-c8ec9b5a289b\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:32:35.342933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.340836 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-model-cache\") pod \"distinct-w0-23843f-kserve-bb7f85f67-wl7fb\" (UID: \"e7673edf-f90d-4902-a858-c8ec9b5a289b\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:32:35.342933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.341145 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-home\") pod \"distinct-w0-23843f-kserve-bb7f85f67-wl7fb\" (UID: \"e7673edf-f90d-4902-a858-c8ec9b5a289b\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:32:35.395936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.379487 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7673edf-f90d-4902-a858-c8ec9b5a289b-tls-certs\") pod \"distinct-w0-23843f-kserve-bb7f85f67-wl7fb\" (UID: \"e7673edf-f90d-4902-a858-c8ec9b5a289b\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:32:35.395936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.381552 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-dshm\") pod \"distinct-w0-23843f-kserve-bb7f85f67-wl7fb\" (UID: \"e7673edf-f90d-4902-a858-c8ec9b5a289b\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:32:35.480982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.475813 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jwwh\" (UniqueName: \"kubernetes.io/projected/e7673edf-f90d-4902-a858-c8ec9b5a289b-kube-api-access-7jwwh\") pod \"distinct-w0-23843f-kserve-bb7f85f67-wl7fb\" (UID: \"e7673edf-f90d-4902-a858-c8ec9b5a289b\") " pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:32:35.506387 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.506351 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:32:35.860950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.859194 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb"] Oct 31 19:32:35.881017 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.879989 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" event={"ID":"78d95c12-a91e-44a2-94a4-001d4c3ec257","Type":"ContainerStarted","Data":"3e4b1a16a778773e4a203fde8c7c68533c8879a4a44d61aeb201879a4a62e8a3"} Oct 31 19:32:35.881017 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.880056 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" event={"ID":"78d95c12-a91e-44a2-94a4-001d4c3ec257","Type":"ContainerStarted","Data":"f39ed49f90664aa9c89c0325bb0f3953bae3208a17a94cd8fdee5e50dced2c54"} Oct 31 19:32:35.886217 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.883629 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" event={"ID":"012ba66a-ce6f-4261-9516-e8f0aa8ff815","Type":"ContainerStarted","Data":"f3dfb43990ecb35499512e9cf0b6372b1b6a9005d56fa8b7666d39f934e48b2b"} Oct 31 19:32:35.886217 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.883692 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" event={"ID":"012ba66a-ce6f-4261-9516-e8f0aa8ff815","Type":"ContainerStarted","Data":"7d13ac87036da33f3e7f7aebd8617a6e4142029c52a76d0d0c59a27f891e0f98"} Oct 31 19:32:35.909855 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.909796 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" podStartSLOduration=2.909776626 podStartE2EDuration="2.909776626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:35.907179611 +0000 UTC m=+790.270000007" watchObservedRunningTime="2026-10-31 19:32:35.909776626 +0000 UTC m=+790.272596998" Oct 31 19:32:35.978979 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:35.978469 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" podStartSLOduration=2.978445905 podStartE2EDuration="2.978445905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:35.951366199 +0000 UTC m=+790.314186593" watchObservedRunningTime="2026-10-31 19:32:35.978445905 +0000 UTC m=+790.341266281" Oct 31 19:32:36.489611 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:36.489581 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:36.490705 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:36.490674 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 31 19:32:36.490818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:36.490734 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" podUID="012ba66a-ce6f-4261-9516-e8f0aa8ff815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 31 19:32:36.566938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:36.565977 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:36.566938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:36.566040 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 31 19:32:36.566938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:36.566096 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" podUID="78d95c12-a91e-44a2-94a4-001d4c3ec257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 31 19:32:36.895783 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:36.895746 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" event={"ID":"e7673edf-f90d-4902-a858-c8ec9b5a289b","Type":"ContainerStarted","Data":"de3d92c5bc16ed668458a0627d3246bd098819fed5feef63598db64c9abc785d"} Oct 31 19:32:36.895998 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:36.895803 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" event={"ID":"e7673edf-f90d-4902-a858-c8ec9b5a289b","Type":"ContainerStarted","Data":"0b1a825d7be0dc8b9936243663f2e553d4d1564042a89cc4b631132ebf4a6a0e"} Oct 31 19:32:36.917103 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:36.917053 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podStartSLOduration=1.917039148 podStartE2EDuration="1.917039148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:36.915950409 +0000 UTC m=+791.278770781" watchObservedRunningTime="2026-10-31 19:32:36.917039148 +0000 UTC m=+791.279859519" Oct 31 19:32:37.490645 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:37.490608 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 31 19:32:37.490976 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:37.490943 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" podUID="012ba66a-ce6f-4261-9516-e8f0aa8ff815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 31 19:32:37.509130 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:37.509097 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:32:37.566937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:37.562250 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 31 19:32:37.566937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:37.562309 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" podUID="78d95c12-a91e-44a2-94a4-001d4c3ec257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 31 19:32:38.489199 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:38.489164 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 31 19:32:38.489419 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:38.489395 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" podUID="012ba66a-ce6f-4261-9516-e8f0aa8ff815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 31 19:32:38.562304 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:38.562265 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 31 19:32:38.562569 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:38.562541 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" podUID="78d95c12-a91e-44a2-94a4-001d4c3ec257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 31 19:32:39.491316 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:39.491279 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 31 19:32:39.491864 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:39.491834 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" podUID="012ba66a-ce6f-4261-9516-e8f0aa8ff815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 31 19:32:39.564936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:39.564873 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 31 19:32:39.565117 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:39.564958 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" podUID="78d95c12-a91e-44a2-94a4-001d4c3ec257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 31 19:32:40.489327 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:40.489287 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 31 19:32:40.489619 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:40.489590 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" podUID="012ba66a-ce6f-4261-9516-e8f0aa8ff815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 31 19:32:40.565938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:40.565299 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 31 19:32:40.565938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:40.565391 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" podUID="78d95c12-a91e-44a2-94a4-001d4c3ec257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 31 19:32:41.489074 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:41.489036 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 31 19:32:41.489251 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:41.489115 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" podUID="012ba66a-ce6f-4261-9516-e8f0aa8ff815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 31 19:32:41.562930 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:41.562350 4910 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 31 19:32:41.562930 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:41.562417 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" podUID="78d95c12-a91e-44a2-94a4-001d4c3ec257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 31 19:32:42.728547 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:32:42.728456 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:32:42.879007 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:42.878973 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:42.879770 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:42.879748 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:42.880830 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:42.880806 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:32:43.052447 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.052373 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:43.053404 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.053309 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:43.053899 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.053881 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 19:32:43.110971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.110934 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n"] Oct 31 19:32:43.115325 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.115295 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:32:43.145685 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.133925 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-23814d\"/\"distinct-2-w3-23814d-kserve-self-signed-certs\"" Oct 31 19:32:43.165985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.162585 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n"] Oct 31 19:32:43.191834 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.191799 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-model-cache\") pod \"distinct-2-w3-23814d-kserve-7588856589-mw99n\" (UID: \"97560889-c009-4292-968a-58758bd2266b\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:32:43.192024 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.191875 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-home\") pod \"distinct-2-w3-23814d-kserve-7588856589-mw99n\" (UID: \"97560889-c009-4292-968a-58758bd2266b\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:32:43.192024 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.191906 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97560889-c009-4292-968a-58758bd2266b-tls-certs\") pod \"distinct-2-w3-23814d-kserve-7588856589-mw99n\" (UID: \"97560889-c009-4292-968a-58758bd2266b\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:32:43.192128 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.192070 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4kdh\" (UniqueName: \"kubernetes.io/projected/97560889-c009-4292-968a-58758bd2266b-kube-api-access-v4kdh\") pod \"distinct-2-w3-23814d-kserve-7588856589-mw99n\" (UID: \"97560889-c009-4292-968a-58758bd2266b\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:32:43.192376 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.192351 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-tmp-dir\") pod \"distinct-2-w3-23814d-kserve-7588856589-mw99n\" (UID: \"97560889-c009-4292-968a-58758bd2266b\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:32:43.192437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.192416 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-dshm\") pod \"distinct-2-w3-23814d-kserve-7588856589-mw99n\" (UID: \"97560889-c009-4292-968a-58758bd2266b\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:32:43.294503 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.294463 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-model-cache\") pod \"distinct-2-w3-23814d-kserve-7588856589-mw99n\" (UID: \"97560889-c009-4292-968a-58758bd2266b\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:32:43.294720 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.294530 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-home\") pod \"distinct-2-w3-23814d-kserve-7588856589-mw99n\" (UID: \"97560889-c009-4292-968a-58758bd2266b\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:32:43.294720 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.294561 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97560889-c009-4292-968a-58758bd2266b-tls-certs\") pod \"distinct-2-w3-23814d-kserve-7588856589-mw99n\" (UID: \"97560889-c009-4292-968a-58758bd2266b\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:32:43.294720 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.294592 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4kdh\" (UniqueName: \"kubernetes.io/projected/97560889-c009-4292-968a-58758bd2266b-kube-api-access-v4kdh\") pod \"distinct-2-w3-23814d-kserve-7588856589-mw99n\" (UID: \"97560889-c009-4292-968a-58758bd2266b\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:32:43.294720 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.294666 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-tmp-dir\") pod \"distinct-2-w3-23814d-kserve-7588856589-mw99n\" (UID: \"97560889-c009-4292-968a-58758bd2266b\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:32:43.294720 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.294693 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-dshm\") pod \"distinct-2-w3-23814d-kserve-7588856589-mw99n\" (UID: \"97560889-c009-4292-968a-58758bd2266b\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:32:43.295059 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.294976 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-model-cache\") pod \"distinct-2-w3-23814d-kserve-7588856589-mw99n\" (UID: \"97560889-c009-4292-968a-58758bd2266b\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:32:43.295391 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.295363 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-tmp-dir\") pod \"distinct-2-w3-23814d-kserve-7588856589-mw99n\" (UID: \"97560889-c009-4292-968a-58758bd2266b\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:32:43.295657 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.295635 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-home\") pod \"distinct-2-w3-23814d-kserve-7588856589-mw99n\" (UID: \"97560889-c009-4292-968a-58758bd2266b\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:32:43.298376 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.298342 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-dshm\") pod \"distinct-2-w3-23814d-kserve-7588856589-mw99n\" (UID: \"97560889-c009-4292-968a-58758bd2266b\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:32:43.299567 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.299544 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97560889-c009-4292-968a-58758bd2266b-tls-certs\") pod \"distinct-2-w3-23814d-kserve-7588856589-mw99n\" (UID: \"97560889-c009-4292-968a-58758bd2266b\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:32:43.308159 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.308124 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4kdh\" (UniqueName: \"kubernetes.io/projected/97560889-c009-4292-968a-58758bd2266b-kube-api-access-v4kdh\") pod \"distinct-2-w3-23814d-kserve-7588856589-mw99n\" (UID: \"97560889-c009-4292-968a-58758bd2266b\") " pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:32:43.478934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.478165 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:32:43.803189 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:32:43.803157 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97560889_c009_4292_968a_58758bd2266b.slice/crio-ceca75d4a3d0dc3a6cb787d7cf131cd72c88296b4b1a9f739fc73d6d4f3f53dd WatchSource:0}: Error finding container ceca75d4a3d0dc3a6cb787d7cf131cd72c88296b4b1a9f739fc73d6d4f3f53dd: Status 404 returned error can't find the container with id ceca75d4a3d0dc3a6cb787d7cf131cd72c88296b4b1a9f739fc73d6d4f3f53dd Oct 31 19:32:43.803761 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:43.803735 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n"] Oct 31 19:32:44.001144 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:44.001100 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" event={"ID":"97560889-c009-4292-968a-58758bd2266b","Type":"ContainerStarted","Data":"88b8a4e0c49d746c34f4a5b5e669d9d53dad08ec4a8c418926e7dfba0e037f0a"} Oct 31 19:32:44.001144 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:44.001143 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" event={"ID":"97560889-c009-4292-968a-58758bd2266b","Type":"ContainerStarted","Data":"ceca75d4a3d0dc3a6cb787d7cf131cd72c88296b4b1a9f739fc73d6d4f3f53dd"} Oct 31 19:32:44.021263 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:44.021210 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podStartSLOduration=1.021198015 podStartE2EDuration="1.021198015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:44.020184275 +0000 UTC m=+798.383004647" watchObservedRunningTime="2026-10-31 19:32:44.021198015 +0000 UTC m=+798.384018386" Oct 31 19:32:44.479251 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:44.479215 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:32:45.116845 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:45.116811 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f75ea166-4h57n"] Oct 31 19:32:45.121292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:45.121264 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75ea166-4h57n" Oct 31 19:32:45.130194 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:45.130168 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:32:45.133073 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:45.130445 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:32:45.138001 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:45.137967 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f75ea166-4h57n"] Oct 31 19:32:45.233431 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:45.233384 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1c443a-03de-4a57-ab05-265775a8ba1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f75ea166-4h57n\" (UID: \"cc1c443a-03de-4a57-ab05-265775a8ba1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75ea166-4h57n" Oct 31 19:32:45.233624 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:45.233521 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cmqt\" (UniqueName: \"kubernetes.io/projected/cc1c443a-03de-4a57-ab05-265775a8ba1a-kube-api-access-2cmqt\") pod \"maas-api-revoke-sub-f75ea166-4h57n\" (UID: \"cc1c443a-03de-4a57-ab05-265775a8ba1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75ea166-4h57n" Oct 31 19:32:45.233691 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:45.233637 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1c443a-03de-4a57-ab05-265775a8ba1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f75ea166-4h57n\" (UID: \"cc1c443a-03de-4a57-ab05-265775a8ba1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75ea166-4h57n" Oct 31 19:32:45.335177 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:45.335139 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1c443a-03de-4a57-ab05-265775a8ba1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f75ea166-4h57n\" (UID: \"cc1c443a-03de-4a57-ab05-265775a8ba1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75ea166-4h57n" Oct 31 19:32:45.335362 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:45.335270 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cmqt\" (UniqueName: \"kubernetes.io/projected/cc1c443a-03de-4a57-ab05-265775a8ba1a-kube-api-access-2cmqt\") pod \"maas-api-revoke-sub-f75ea166-4h57n\" (UID: \"cc1c443a-03de-4a57-ab05-265775a8ba1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75ea166-4h57n" Oct 31 19:32:45.335427 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:45.335407 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1c443a-03de-4a57-ab05-265775a8ba1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f75ea166-4h57n\" (UID: \"cc1c443a-03de-4a57-ab05-265775a8ba1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75ea166-4h57n" Oct 31 19:32:45.335992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:45.335958 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1c443a-03de-4a57-ab05-265775a8ba1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f75ea166-4h57n\" (UID: \"cc1c443a-03de-4a57-ab05-265775a8ba1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75ea166-4h57n" Oct 31 19:32:45.337957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:45.337896 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1c443a-03de-4a57-ab05-265775a8ba1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f75ea166-4h57n\" (UID: \"cc1c443a-03de-4a57-ab05-265775a8ba1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75ea166-4h57n" Oct 31 19:32:45.349339 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:45.349310 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cmqt\" (UniqueName: \"kubernetes.io/projected/cc1c443a-03de-4a57-ab05-265775a8ba1a-kube-api-access-2cmqt\") pod \"maas-api-revoke-sub-f75ea166-4h57n\" (UID: \"cc1c443a-03de-4a57-ab05-265775a8ba1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75ea166-4h57n" Oct 31 19:32:45.463687 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:45.463600 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75ea166-4h57n" Oct 31 19:32:45.507182 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:45.507150 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:32:45.531495 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:45.531444 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:32:45.765995 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:45.765901 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f75ea166-4h57n"] Oct 31 19:32:46.024798 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:46.024701 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75ea166-4h57n" event={"ID":"cc1c443a-03de-4a57-ab05-265775a8ba1a","Type":"ContainerStarted","Data":"f643f24d9fd25387b7524c482542b9db023e510448b37c117337ebf2bcce7209"} Oct 31 19:32:46.036728 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:46.036698 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:32:47.034534 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:47.034493 4910 generic.go:356] "Generic (PLEG): container finished" podID="cc1c443a-03de-4a57-ab05-265775a8ba1a" containerID="e56d6038808de5de60e3f3681138d46d18a04f5efe56133aa0a05ef04a226f05" exitCode=0 Oct 31 19:32:47.035030 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:47.034575 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75ea166-4h57n" event={"ID":"cc1c443a-03de-4a57-ab05-265775a8ba1a","Type":"ContainerDied","Data":"e56d6038808de5de60e3f3681138d46d18a04f5efe56133aa0a05ef04a226f05"} Oct 31 19:32:48.316938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.307202 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf"] Oct 31 19:32:48.334121 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.329163 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.449295 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.449256 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf"] Oct 31 19:32:48.546096 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.546052 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.546439 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.546412 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.546727 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.546707 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.606381 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.598150 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxjjs\" (UniqueName: \"kubernetes.io/projected/005cc55c-8e13-47d1-aa18-ddb1fb79432c-kube-api-access-dxjjs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.606381 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.598211 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.606381 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.598318 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.606381 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.598382 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.606381 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.598450 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.606381 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.598492 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.708850 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.708806 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.709232 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.709205 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.709402 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.709385 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxjjs\" (UniqueName: \"kubernetes.io/projected/005cc55c-8e13-47d1-aa18-ddb1fb79432c-kube-api-access-dxjjs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.709534 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.709515 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.709700 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.709685 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.709844 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.709827 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.710028 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.710009 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.710233 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.710215 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.710461 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.710443 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.711063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.711041 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.711435 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.711416 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.711784 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.711767 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.712854 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.712833 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.729256 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.713461 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.745693 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.741302 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.745693 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.742291 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxjjs\" (UniqueName: \"kubernetes.io/projected/005cc55c-8e13-47d1-aa18-ddb1fb79432c-kube-api-access-dxjjs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.746333 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.746307 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.760263 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.760204 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:48.828595 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.828276 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75ea166-4h57n" Oct 31 19:32:48.849961 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.840772 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v"] Oct 31 19:32:48.849961 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.841825 4910 cpu_manager.go:414] "containerMap: removing container" podUID="cc1c443a-03de-4a57-ab05-265775a8ba1a" containerName="revoke-keys" Oct 31 19:32:48.849961 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.841847 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc1c443a-03de-4a57-ab05-265775a8ba1a" containerName="revoke-keys" Oct 31 19:32:48.849961 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.842123 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc1c443a-03de-4a57-ab05-265775a8ba1a" containerName="revoke-keys" Oct 31 19:32:48.852206 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.852179 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:48.896061 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.895976 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v"] Oct 31 19:32:48.931742 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.931680 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc1c443a-03de-4a57-ab05-265775a8ba1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1c443a-03de-4a57-ab05-265775a8ba1a-maas-api-openshift-ca\") pod \"cc1c443a-03de-4a57-ab05-265775a8ba1a\" (UID: \"cc1c443a-03de-4a57-ab05-265775a8ba1a\") " Oct 31 19:32:48.932102 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.932059 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc1c443a-03de-4a57-ab05-265775a8ba1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1c443a-03de-4a57-ab05-265775a8ba1a-maas-api-xks-ca\") pod \"cc1c443a-03de-4a57-ab05-265775a8ba1a\" (UID: \"cc1c443a-03de-4a57-ab05-265775a8ba1a\") " Oct 31 19:32:48.932491 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.932422 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc1c443a-03de-4a57-ab05-265775a8ba1a-kube-api-access-2cmqt\" (UniqueName: \"kubernetes.io/projected/cc1c443a-03de-4a57-ab05-265775a8ba1a-kube-api-access-2cmqt\") pod \"cc1c443a-03de-4a57-ab05-265775a8ba1a\" (UID: \"cc1c443a-03de-4a57-ab05-265775a8ba1a\") " Oct 31 19:32:48.934310 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:48.934286 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1c443a-03de-4a57-ab05-265775a8ba1a-maas-api-openshift-ca" pod "cc1c443a-03de-4a57-ab05-265775a8ba1a" (UID: "cc1c443a-03de-4a57-ab05-265775a8ba1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:32:49.032683 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.032227 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:49.032954 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.032902 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1c443a-03de-4a57-ab05-265775a8ba1a-kube-api-access-2cmqt" pod "cc1c443a-03de-4a57-ab05-265775a8ba1a" (UID: "cc1c443a-03de-4a57-ab05-265775a8ba1a"). InnerVolumeSpecName "kube-api-access-2cmqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:32:49.068676 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.068623 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.068960 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.068905 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a24bfa8-34c0-4825-876a-c2fec6019829-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.092338 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.092308 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.092601 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.092560 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.092765 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.092741 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.092966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.092948 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.093170 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.093154 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.093292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.093277 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtq2d\" (UniqueName: \"kubernetes.io/projected/0a24bfa8-34c0-4825-876a-c2fec6019829-kube-api-access-mtq2d\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.234991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.232759 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.234991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.232954 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc1c443a-03de-4a57-ab05-265775a8ba1a-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:32:49.234991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.232979 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cmqt\" (UniqueName: \"kubernetes.io/projected/cc1c443a-03de-4a57-ab05-265775a8ba1a-kube-api-access-2cmqt\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:32:49.234991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.199327 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75ea166-4h57n" Oct 31 19:32:49.234991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.199248 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f75ea166-4h57n" event={"ID":"cc1c443a-03de-4a57-ab05-265775a8ba1a","Type":"ContainerDied","Data":"f643f24d9fd25387b7524c482542b9db023e510448b37c117337ebf2bcce7209"} Oct 31 19:32:49.234991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.233229 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f643f24d9fd25387b7524c482542b9db023e510448b37c117337ebf2bcce7209" Oct 31 19:32:49.235598 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.235555 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc1c443a-03de-4a57-ab05-265775a8ba1a-maas-api-xks-ca" pod "cc1c443a-03de-4a57-ab05-265775a8ba1a" (UID: "cc1c443a-03de-4a57-ab05-265775a8ba1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:32:49.352935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.350862 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.352935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.350943 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a24bfa8-34c0-4825-876a-c2fec6019829-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.352935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.351072 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.352935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.351132 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.352935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.351179 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.352935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.351257 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.352935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.351327 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.352935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.351355 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtq2d\" (UniqueName: \"kubernetes.io/projected/0a24bfa8-34c0-4825-876a-c2fec6019829-kube-api-access-mtq2d\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.352935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.351410 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.352935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.351518 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc1c443a-03de-4a57-ab05-265775a8ba1a-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:32:49.384770 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.384612 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.388282 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.388223 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.414936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.388654 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.414936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.388928 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.414936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.398874 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.414936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.399721 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.414936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.401045 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a24bfa8-34c0-4825-876a-c2fec6019829-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.418934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.416720 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtq2d\" (UniqueName: \"kubernetes.io/projected/0a24bfa8-34c0-4825-876a-c2fec6019829-kube-api-access-mtq2d\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.418934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.417246 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.751933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.749182 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:49.856447 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:49.803367 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:50.086843 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:50.086201 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 19:32:50.176417 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:50.158033 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:50.200000 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:50.195338 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:50.200000 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:50.195437 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:50.200000 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:50.195468 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf"] Oct 31 19:32:50.200000 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:50.195487 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:50.495431 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:50.494963 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:32:50.495431 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:50.495025 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" podUID="16761f60-662e-4eb1-a247-b00378149cc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:32:50.524972 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:50.524822 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" event={"ID":"005cc55c-8e13-47d1-aa18-ddb1fb79432c","Type":"ContainerStarted","Data":"d89da33412fe1b97965bd3ac6c77bb042e657a50b974746a384760d49ae18243"} Oct 31 19:32:50.750965 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:50.750935 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v"] Oct 31 19:32:50.761949 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:50.761900 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:50.762060 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:50.761999 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:50.762060 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:50.762045 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:32:50.891906 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:50.891816 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6ca1252-5nwbd"] Oct 31 19:32:50.896944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:50.896906 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ca1252-5nwbd" Oct 31 19:32:50.909690 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:50.906143 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:32:50.909690 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:50.906270 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:32:50.909690 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:50.908817 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6ca1252-5nwbd"] Oct 31 19:32:51.037170 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:51.032106 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmrmh\" (UniqueName: \"kubernetes.io/projected/a2056a6d-547a-4aeb-b054-ca3ceb602d4f-kube-api-access-mmrmh\") pod \"maas-api-revoke-sub-a6ca1252-5nwbd\" (UID: \"a2056a6d-547a-4aeb-b054-ca3ceb602d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ca1252-5nwbd" Oct 31 19:32:51.037170 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:51.032344 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2056a6d-547a-4aeb-b054-ca3ceb602d4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6ca1252-5nwbd\" (UID: \"a2056a6d-547a-4aeb-b054-ca3ceb602d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ca1252-5nwbd" Oct 31 19:32:51.037170 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:51.032475 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2056a6d-547a-4aeb-b054-ca3ceb602d4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6ca1252-5nwbd\" (UID: \"a2056a6d-547a-4aeb-b054-ca3ceb602d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ca1252-5nwbd" Oct 31 19:32:51.135598 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:51.133986 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2056a6d-547a-4aeb-b054-ca3ceb602d4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6ca1252-5nwbd\" (UID: \"a2056a6d-547a-4aeb-b054-ca3ceb602d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ca1252-5nwbd" Oct 31 19:32:51.135598 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:51.134147 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2056a6d-547a-4aeb-b054-ca3ceb602d4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6ca1252-5nwbd\" (UID: \"a2056a6d-547a-4aeb-b054-ca3ceb602d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ca1252-5nwbd" Oct 31 19:32:51.135598 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:51.134197 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmrmh\" (UniqueName: \"kubernetes.io/projected/a2056a6d-547a-4aeb-b054-ca3ceb602d4f-kube-api-access-mmrmh\") pod \"maas-api-revoke-sub-a6ca1252-5nwbd\" (UID: \"a2056a6d-547a-4aeb-b054-ca3ceb602d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ca1252-5nwbd" Oct 31 19:32:51.135598 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:51.135277 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2056a6d-547a-4aeb-b054-ca3ceb602d4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6ca1252-5nwbd\" (UID: \"a2056a6d-547a-4aeb-b054-ca3ceb602d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ca1252-5nwbd" Oct 31 19:32:51.138215 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:51.137906 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2056a6d-547a-4aeb-b054-ca3ceb602d4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6ca1252-5nwbd\" (UID: \"a2056a6d-547a-4aeb-b054-ca3ceb602d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ca1252-5nwbd" Oct 31 19:32:51.149237 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:51.149208 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmrmh\" (UniqueName: \"kubernetes.io/projected/a2056a6d-547a-4aeb-b054-ca3ceb602d4f-kube-api-access-mmrmh\") pod \"maas-api-revoke-sub-a6ca1252-5nwbd\" (UID: \"a2056a6d-547a-4aeb-b054-ca3ceb602d4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ca1252-5nwbd" Oct 31 19:32:51.234731 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:51.234693 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ca1252-5nwbd" Oct 31 19:32:51.563273 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:32:51.563241 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2056a6d_547a_4aeb_b054_ca3ceb602d4f.slice/crio-833c93a45c5b35d3ca7fda8a809f1c4b793030c60d276b748e30eb7581d9a466 WatchSource:0}: Error finding container 833c93a45c5b35d3ca7fda8a809f1c4b793030c60d276b748e30eb7581d9a466: Status 404 returned error can't find the container with id 833c93a45c5b35d3ca7fda8a809f1c4b793030c60d276b748e30eb7581d9a466 Oct 31 19:32:51.569759 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:51.569729 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6ca1252-5nwbd"] Oct 31 19:32:51.571998 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:51.571967 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" event={"ID":"0a24bfa8-34c0-4825-876a-c2fec6019829","Type":"ContainerStarted","Data":"28664d6bcfb9b6b8214d9dd2d1320f7bd7de6dd2f021a361d2db9d1085eab3e1"} Oct 31 19:32:51.572112 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:51.572026 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" event={"ID":"0a24bfa8-34c0-4825-876a-c2fec6019829","Type":"ContainerStarted","Data":"9ae8891146d4c577883c034964c177ff661d0691c96f323176465511c986d589"} Oct 31 19:32:51.574447 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:51.574413 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" event={"ID":"005cc55c-8e13-47d1-aa18-ddb1fb79432c","Type":"ContainerStarted","Data":"a0ec8840783025046c13d0a7b093c07e4ff6fc0d2810bf63476f2685c21eaacf"} Oct 31 19:32:51.598786 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:51.598730 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" podStartSLOduration=3.5987125840000003 podStartE2EDuration="3.598712584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:51.597240294 +0000 UTC m=+805.960060666" watchObservedRunningTime="2026-10-31 19:32:51.598712584 +0000 UTC m=+805.961532958" Oct 31 19:32:51.661939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:51.661264 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" podStartSLOduration=3.661245971 podStartE2EDuration="3.661245971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:51.656024286 +0000 UTC m=+806.018844657" watchObservedRunningTime="2026-10-31 19:32:51.661245971 +0000 UTC m=+806.024066343" Oct 31 19:32:51.762179 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:51.759343 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 31 19:32:51.762179 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:51.759396 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" podUID="0a24bfa8-34c0-4825-876a-c2fec6019829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 31 19:32:51.771187 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:51.771161 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:52.059934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:52.047954 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:52.059934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:52.050845 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 31 19:32:52.059934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:52.050940 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" podUID="005cc55c-8e13-47d1-aa18-ddb1fb79432c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 31 19:32:52.592962 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:52.592501 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ca1252-5nwbd" event={"ID":"a2056a6d-547a-4aeb-b054-ca3ceb602d4f","Type":"ContainerStarted","Data":"cd4d7f26c0e8d7e8b79e6da4fcf702f4de112c483efead0c0463e063413fcf33"} Oct 31 19:32:52.592962 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:52.592581 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ca1252-5nwbd" event={"ID":"a2056a6d-547a-4aeb-b054-ca3ceb602d4f","Type":"ContainerStarted","Data":"833c93a45c5b35d3ca7fda8a809f1c4b793030c60d276b748e30eb7581d9a466"} Oct 31 19:32:52.616786 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:52.616727 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ca1252-5nwbd" podStartSLOduration=2.616708524 podStartE2EDuration="2.616708524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:52.612692922 +0000 UTC m=+806.975513294" watchObservedRunningTime="2026-10-31 19:32:52.616708524 +0000 UTC m=+806.979528896" Oct 31 19:32:52.759331 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:52.758256 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 31 19:32:52.759331 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:52.758308 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" podUID="0a24bfa8-34c0-4825-876a-c2fec6019829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 31 19:32:53.043041 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.042250 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 31 19:32:53.043041 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.042312 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" podUID="005cc55c-8e13-47d1-aa18-ddb1fb79432c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 31 19:32:53.206183 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.206139 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt"] Oct 31 19:32:53.214248 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.214222 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:32:53.216810 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.216767 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt"] Oct 31 19:32:53.234943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.234547 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-996c32\"/\"distinct-2-w4-996c32-kserve-self-signed-certs\"" Oct 31 19:32:53.604942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.604625 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:32:53.688970 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.656466 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-model-cache\") pod \"distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt\" (UID: \"97abf78a-128a-4f23-947f-284d7dfecace\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:32:53.688970 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.656559 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-home\") pod \"distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt\" (UID: \"97abf78a-128a-4f23-947f-284d7dfecace\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:32:53.688970 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.656623 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97abf78a-128a-4f23-947f-284d7dfecace-tls-certs\") pod \"distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt\" (UID: \"97abf78a-128a-4f23-947f-284d7dfecace\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:32:53.688970 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.656743 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-tmp-dir\") pod \"distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt\" (UID: \"97abf78a-128a-4f23-947f-284d7dfecace\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:32:53.688970 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.656897 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jszf\" (UniqueName: \"kubernetes.io/projected/97abf78a-128a-4f23-947f-284d7dfecace-kube-api-access-9jszf\") pod \"distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt\" (UID: \"97abf78a-128a-4f23-947f-284d7dfecace\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:32:53.688970 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.657028 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-dshm\") pod \"distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt\" (UID: \"97abf78a-128a-4f23-947f-284d7dfecace\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:32:53.765935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.765038 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-dshm\") pod \"distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt\" (UID: \"97abf78a-128a-4f23-947f-284d7dfecace\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:32:53.765935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.765246 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-model-cache\") pod \"distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt\" (UID: \"97abf78a-128a-4f23-947f-284d7dfecace\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:32:53.765935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.765321 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-home\") pod \"distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt\" (UID: \"97abf78a-128a-4f23-947f-284d7dfecace\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:32:53.765935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.765391 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97abf78a-128a-4f23-947f-284d7dfecace-tls-certs\") pod \"distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt\" (UID: \"97abf78a-128a-4f23-947f-284d7dfecace\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:32:53.765935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.765570 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-tmp-dir\") pod \"distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt\" (UID: \"97abf78a-128a-4f23-947f-284d7dfecace\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:32:53.765935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.765711 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jszf\" (UniqueName: \"kubernetes.io/projected/97abf78a-128a-4f23-947f-284d7dfecace-kube-api-access-9jszf\") pod \"distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt\" (UID: \"97abf78a-128a-4f23-947f-284d7dfecace\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:32:53.787015 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.783505 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-model-cache\") pod \"distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt\" (UID: \"97abf78a-128a-4f23-947f-284d7dfecace\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:32:53.787015 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.784622 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-home\") pod \"distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt\" (UID: \"97abf78a-128a-4f23-947f-284d7dfecace\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:32:53.787015 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.784893 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-tmp-dir\") pod \"distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt\" (UID: \"97abf78a-128a-4f23-947f-284d7dfecace\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:32:53.828959 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.796669 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 31 19:32:53.828959 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.824109 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" podUID="0a24bfa8-34c0-4825-876a-c2fec6019829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 31 19:32:53.848696 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.848662 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jszf\" (UniqueName: \"kubernetes.io/projected/97abf78a-128a-4f23-947f-284d7dfecace-kube-api-access-9jszf\") pod \"distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt\" (UID: \"97abf78a-128a-4f23-947f-284d7dfecace\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:32:53.865989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.859246 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-dshm\") pod \"distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt\" (UID: \"97abf78a-128a-4f23-947f-284d7dfecace\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:32:53.867077 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.865972 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97abf78a-128a-4f23-947f-284d7dfecace-tls-certs\") pod \"distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt\" (UID: \"97abf78a-128a-4f23-947f-284d7dfecace\") " pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:32:53.908064 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.880905 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:32:53.908628 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.908599 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:32:53.911110 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.910871 4910 generic.go:356] "Generic (PLEG): container finished" podID="a2056a6d-547a-4aeb-b054-ca3ceb602d4f" containerID="cd4d7f26c0e8d7e8b79e6da4fcf702f4de112c483efead0c0463e063413fcf33" exitCode=0 Oct 31 19:32:53.911240 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:53.911117 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ca1252-5nwbd" event={"ID":"a2056a6d-547a-4aeb-b054-ca3ceb602d4f","Type":"ContainerDied","Data":"cd4d7f26c0e8d7e8b79e6da4fcf702f4de112c483efead0c0463e063413fcf33"} Oct 31 19:32:54.053857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:54.053381 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 31 19:32:54.053857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:54.053477 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" podUID="005cc55c-8e13-47d1-aa18-ddb1fb79432c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 31 19:32:54.507692 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:54.507656 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:32:54.599174 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:54.586007 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt"] Oct 31 19:32:54.603264 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:32:54.603233 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97abf78a_128a_4f23_947f_284d7dfecace.slice/crio-ff47317338a28fd8598fba6b20fac757cd28bbf375ee09b8d3ab41a04a090ee2 WatchSource:0}: Error finding container ff47317338a28fd8598fba6b20fac757cd28bbf375ee09b8d3ab41a04a090ee2: Status 404 returned error can't find the container with id ff47317338a28fd8598fba6b20fac757cd28bbf375ee09b8d3ab41a04a090ee2 Oct 31 19:32:54.774000 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:54.762127 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 31 19:32:54.774000 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:54.762190 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" podUID="0a24bfa8-34c0-4825-876a-c2fec6019829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 31 19:32:54.960584 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:54.960549 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" event={"ID":"97abf78a-128a-4f23-947f-284d7dfecace","Type":"ContainerStarted","Data":"747142fb9c733d12c2b3edf18268ae5a375df36c7b9927f79472a39c9aed5bde"} Oct 31 19:32:54.960825 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:54.960799 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" event={"ID":"97abf78a-128a-4f23-947f-284d7dfecace","Type":"ContainerStarted","Data":"ff47317338a28fd8598fba6b20fac757cd28bbf375ee09b8d3ab41a04a090ee2"} Oct 31 19:32:54.991242 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:54.991165 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podStartSLOduration=1.9911459219999998 podStartE2EDuration="1.991145922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:54.988097453 +0000 UTC m=+809.350917827" watchObservedRunningTime="2026-10-31 19:32:54.991145922 +0000 UTC m=+809.353966296" Oct 31 19:32:55.046993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:55.046310 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 31 19:32:55.046993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:55.046373 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" podUID="005cc55c-8e13-47d1-aa18-ddb1fb79432c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 31 19:32:55.511312 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:55.511276 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ca1252-5nwbd" Oct 31 19:32:55.552951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:55.528472 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2056a6d-547a-4aeb-b054-ca3ceb602d4f-kube-api-access-mmrmh\" (UniqueName: \"kubernetes.io/projected/a2056a6d-547a-4aeb-b054-ca3ceb602d4f-kube-api-access-mmrmh\") pod \"a2056a6d-547a-4aeb-b054-ca3ceb602d4f\" (UID: \"a2056a6d-547a-4aeb-b054-ca3ceb602d4f\") " Oct 31 19:32:55.552951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:55.528536 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2056a6d-547a-4aeb-b054-ca3ceb602d4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2056a6d-547a-4aeb-b054-ca3ceb602d4f-maas-api-xks-ca\") pod \"a2056a6d-547a-4aeb-b054-ca3ceb602d4f\" (UID: \"a2056a6d-547a-4aeb-b054-ca3ceb602d4f\") " Oct 31 19:32:55.552951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:55.528807 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2056a6d-547a-4aeb-b054-ca3ceb602d4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2056a6d-547a-4aeb-b054-ca3ceb602d4f-maas-api-openshift-ca\") pod \"a2056a6d-547a-4aeb-b054-ca3ceb602d4f\" (UID: \"a2056a6d-547a-4aeb-b054-ca3ceb602d4f\") " Oct 31 19:32:55.570821 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:55.558878 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2056a6d-547a-4aeb-b054-ca3ceb602d4f-maas-api-openshift-ca" pod "a2056a6d-547a-4aeb-b054-ca3ceb602d4f" (UID: "a2056a6d-547a-4aeb-b054-ca3ceb602d4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:32:55.640935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:55.637409 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2056a6d-547a-4aeb-b054-ca3ceb602d4f-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:32:55.675865 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:55.675755 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2056a6d-547a-4aeb-b054-ca3ceb602d4f-kube-api-access-mmrmh" pod "a2056a6d-547a-4aeb-b054-ca3ceb602d4f" (UID: "a2056a6d-547a-4aeb-b054-ca3ceb602d4f"). InnerVolumeSpecName "kube-api-access-mmrmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:32:55.679139 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:55.679110 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2056a6d-547a-4aeb-b054-ca3ceb602d4f-maas-api-xks-ca" pod "a2056a6d-547a-4aeb-b054-ca3ceb602d4f" (UID: "a2056a6d-547a-4aeb-b054-ca3ceb602d4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:32:55.781266 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:55.781224 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmrmh\" (UniqueName: \"kubernetes.io/projected/a2056a6d-547a-4aeb-b054-ca3ceb602d4f-kube-api-access-mmrmh\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:32:55.792884 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:55.792852 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2056a6d-547a-4aeb-b054-ca3ceb602d4f-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:32:55.795241 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:55.789048 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 31 19:32:55.795400 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:55.795373 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" podUID="0a24bfa8-34c0-4825-876a-c2fec6019829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 31 19:32:55.909928 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:55.909872 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:32:55.993984 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:55.993640 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ca1252-5nwbd" Oct 31 19:32:55.997645 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:55.997051 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ca1252-5nwbd" event={"ID":"a2056a6d-547a-4aeb-b054-ca3ceb602d4f","Type":"ContainerDied","Data":"833c93a45c5b35d3ca7fda8a809f1c4b793030c60d276b748e30eb7581d9a466"} Oct 31 19:32:55.997645 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:55.997128 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="833c93a45c5b35d3ca7fda8a809f1c4b793030c60d276b748e30eb7581d9a466" Oct 31 19:32:56.043993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:56.043959 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 31 19:32:56.044271 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:56.044239 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" podUID="005cc55c-8e13-47d1-aa18-ddb1fb79432c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 31 19:32:56.808102 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:56.808066 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 31 19:32:56.808676 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:56.808640 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" podUID="0a24bfa8-34c0-4825-876a-c2fec6019829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 31 19:32:57.051942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:57.050027 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 31 19:32:57.051942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:57.050090 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" podUID="005cc55c-8e13-47d1-aa18-ddb1fb79432c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 31 19:32:57.757363 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:57.757323 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 31 19:32:57.757689 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:57.757654 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" podUID="0a24bfa8-34c0-4825-876a-c2fec6019829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 31 19:32:58.041867 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:58.041775 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 31 19:32:58.041867 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:58.041846 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" podUID="005cc55c-8e13-47d1-aa18-ddb1fb79432c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 31 19:32:58.868661 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:58.866259 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:58.868661 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:58.867925 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:58.868958 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:58.868889 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:32:59.052350 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:59.052317 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:59.053044 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:59.053024 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:59.053633 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:59.053613 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:32:59.126982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:59.126857 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9491c70-m9xkz"] Oct 31 19:32:59.130927 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:59.127783 4910 cpu_manager.go:414] "containerMap: removing container" podUID="a2056a6d-547a-4aeb-b054-ca3ceb602d4f" containerName="revoke-keys" Oct 31 19:32:59.130927 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:59.127810 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2056a6d-547a-4aeb-b054-ca3ceb602d4f" containerName="revoke-keys" Oct 31 19:32:59.130927 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:59.128060 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2056a6d-547a-4aeb-b054-ca3ceb602d4f" containerName="revoke-keys" Oct 31 19:32:59.155950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:59.133086 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9491c70-m9xkz" Oct 31 19:32:59.155950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:59.137491 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:32:59.155950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:59.137589 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:32:59.155950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:59.154353 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9491c70-m9xkz"] Oct 31 19:32:59.234071 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:59.232654 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9491c70-m9xkz\" (UID: \"ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9491c70-m9xkz" Oct 31 19:32:59.234071 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:59.232708 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9491c70-m9xkz\" (UID: \"ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9491c70-m9xkz" Oct 31 19:32:59.234071 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:59.232973 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gzj5\" (UniqueName: \"kubernetes.io/projected/ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0-kube-api-access-7gzj5\") pod \"maas-api-revoke-sub-e9491c70-m9xkz\" (UID: \"ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9491c70-m9xkz" Oct 31 19:32:59.334306 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:59.334266 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9491c70-m9xkz\" (UID: \"ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9491c70-m9xkz" Oct 31 19:32:59.334510 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:59.334322 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9491c70-m9xkz\" (UID: \"ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9491c70-m9xkz" Oct 31 19:32:59.334591 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:59.334526 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gzj5\" (UniqueName: \"kubernetes.io/projected/ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0-kube-api-access-7gzj5\") pod \"maas-api-revoke-sub-e9491c70-m9xkz\" (UID: \"ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9491c70-m9xkz" Oct 31 19:32:59.335566 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:59.335535 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9491c70-m9xkz\" (UID: \"ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9491c70-m9xkz" Oct 31 19:32:59.337296 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:59.337268 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9491c70-m9xkz\" (UID: \"ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9491c70-m9xkz" Oct 31 19:32:59.347359 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:59.347332 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gzj5\" (UniqueName: \"kubernetes.io/projected/ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0-kube-api-access-7gzj5\") pod \"maas-api-revoke-sub-e9491c70-m9xkz\" (UID: \"ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9491c70-m9xkz" Oct 31 19:32:59.464419 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:59.464325 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9491c70-m9xkz" Oct 31 19:32:59.772058 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:32:59.772012 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded2cdc6d_4017_4a4b_baf3_7726ba94e1b0.slice/crio-d0f79e4a9f4ca1563bb26887c311457c80654397141ace21fe348767477dd695 WatchSource:0}: Error finding container d0f79e4a9f4ca1563bb26887c311457c80654397141ace21fe348767477dd695: Status 404 returned error can't find the container with id d0f79e4a9f4ca1563bb26887c311457c80654397141ace21fe348767477dd695 Oct 31 19:32:59.774617 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:32:59.774469 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9491c70-m9xkz"] Oct 31 19:33:00.036295 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:00.036253 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9491c70-m9xkz" event={"ID":"ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0","Type":"ContainerStarted","Data":"d0f79e4a9f4ca1563bb26887c311457c80654397141ace21fe348767477dd695"} Oct 31 19:33:01.047652 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:01.047560 4910 generic.go:356] "Generic (PLEG): container finished" podID="ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0" containerID="a0660493712715953240af4379666ebd0774d303541ea8bd2f868bc1ddf78f5e" exitCode=0 Oct 31 19:33:01.048065 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:01.047781 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9491c70-m9xkz" event={"ID":"ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0","Type":"ContainerDied","Data":"a0660493712715953240af4379666ebd0774d303541ea8bd2f868bc1ddf78f5e"} Oct 31 19:33:02.310473 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:02.310440 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9491c70-m9xkz" Oct 31 19:33:02.381866 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:02.381836 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0-maas-api-xks-ca\") pod \"ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0\" (UID: \"ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0\") " Oct 31 19:33:02.382080 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:02.382063 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0-maas-api-openshift-ca\") pod \"ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0\" (UID: \"ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0\") " Oct 31 19:33:02.382136 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:02.382124 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0-kube-api-access-7gzj5\" (UniqueName: \"kubernetes.io/projected/ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0-kube-api-access-7gzj5\") pod \"ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0\" (UID: \"ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0\") " Oct 31 19:33:02.382424 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:02.382386 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0-maas-api-openshift-ca" pod "ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0" (UID: "ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:33:02.382841 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:02.382823 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:33:02.383980 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:02.383955 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0-maas-api-xks-ca" pod "ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0" (UID: "ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:33:02.384111 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:02.384089 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0-kube-api-access-7gzj5" pod "ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0" (UID: "ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0"). InnerVolumeSpecName "kube-api-access-7gzj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:33:02.484496 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:02.484463 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gzj5\" (UniqueName: \"kubernetes.io/projected/ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0-kube-api-access-7gzj5\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:33:02.484496 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:02.484491 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:33:02.915160 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:02.915123 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl"] Oct 31 19:33:02.915786 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:02.915770 4910 cpu_manager.go:414] "containerMap: removing container" podUID="ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0" containerName="revoke-keys" Oct 31 19:33:02.915870 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:02.915786 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0" containerName="revoke-keys" Oct 31 19:33:02.915948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:02.915878 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0" containerName="revoke-keys" Oct 31 19:33:02.919843 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:02.919819 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:02.970529 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:02.969967 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl"] Oct 31 19:33:03.001957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:02.992216 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.001957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:02.992271 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e946922-c762-4b9d-93af-73ab4af9fd50-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.001957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:02.992338 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.001957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:02.992379 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bft7j\" (UniqueName: \"kubernetes.io/projected/2e946922-c762-4b9d-93af-73ab4af9fd50-kube-api-access-bft7j\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.001957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:02.992421 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.001957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:02.992543 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.001957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:02.992599 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.001957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:02.992659 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.001957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:02.992728 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.070852 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.070813 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9491c70-m9xkz" event={"ID":"ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0","Type":"ContainerDied","Data":"d0f79e4a9f4ca1563bb26887c311457c80654397141ace21fe348767477dd695"} Oct 31 19:33:03.071031 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.070883 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0f79e4a9f4ca1563bb26887c311457c80654397141ace21fe348767477dd695" Oct 31 19:33:03.071031 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.070830 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9491c70-m9xkz" Oct 31 19:33:03.094730 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.094700 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.094887 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.094803 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.094887 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.094842 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.094887 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.094863 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.095082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.094949 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.095139 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.095109 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.095204 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.095140 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e946922-c762-4b9d-93af-73ab4af9fd50-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.095267 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.095203 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.095267 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.095234 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.095267 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.095245 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bft7j\" (UniqueName: \"kubernetes.io/projected/2e946922-c762-4b9d-93af-73ab4af9fd50-kube-api-access-bft7j\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.095430 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.095204 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.095481 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.095429 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.095695 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.095658 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.095862 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.095842 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e946922-c762-4b9d-93af-73ab4af9fd50-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.097482 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.097462 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.097611 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.097590 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.107639 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.107612 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.107945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.107903 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bft7j\" (UniqueName: \"kubernetes.io/projected/2e946922-c762-4b9d-93af-73ab4af9fd50-kube-api-access-bft7j\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.252841 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.252751 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:03.287519 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.287488 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-150e73ff-62hcr"] Oct 31 19:33:03.293704 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.293678 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150e73ff-62hcr" Oct 31 19:33:03.300377 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.300159 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:33:03.300377 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.300286 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:33:03.301568 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.301448 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-150e73ff-62hcr"] Oct 31 19:33:03.402929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.402879 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a49b819-0081-411c-87d4-8a703c754708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-150e73ff-62hcr\" (UID: \"8a49b819-0081-411c-87d4-8a703c754708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150e73ff-62hcr" Oct 31 19:33:03.403313 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.403161 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a49b819-0081-411c-87d4-8a703c754708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-150e73ff-62hcr\" (UID: \"8a49b819-0081-411c-87d4-8a703c754708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150e73ff-62hcr" Oct 31 19:33:03.403313 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.403206 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hjlm\" (UniqueName: \"kubernetes.io/projected/8a49b819-0081-411c-87d4-8a703c754708-kube-api-access-9hjlm\") pod \"maas-api-revoke-sub-150e73ff-62hcr\" (UID: \"8a49b819-0081-411c-87d4-8a703c754708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150e73ff-62hcr" Oct 31 19:33:03.491963 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.491059 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr"] Oct 31 19:33:03.499135 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.499108 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.505785 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.505711 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a49b819-0081-411c-87d4-8a703c754708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-150e73ff-62hcr\" (UID: \"8a49b819-0081-411c-87d4-8a703c754708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150e73ff-62hcr" Oct 31 19:33:03.505963 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.505947 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a49b819-0081-411c-87d4-8a703c754708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-150e73ff-62hcr\" (UID: \"8a49b819-0081-411c-87d4-8a703c754708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150e73ff-62hcr" Oct 31 19:33:03.506046 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.506001 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hjlm\" (UniqueName: \"kubernetes.io/projected/8a49b819-0081-411c-87d4-8a703c754708-kube-api-access-9hjlm\") pod \"maas-api-revoke-sub-150e73ff-62hcr\" (UID: \"8a49b819-0081-411c-87d4-8a703c754708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150e73ff-62hcr" Oct 31 19:33:03.515508 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.514003 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a49b819-0081-411c-87d4-8a703c754708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-150e73ff-62hcr\" (UID: \"8a49b819-0081-411c-87d4-8a703c754708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150e73ff-62hcr" Oct 31 19:33:03.515508 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.514759 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr"] Oct 31 19:33:03.517260 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.517236 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a49b819-0081-411c-87d4-8a703c754708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-150e73ff-62hcr\" (UID: \"8a49b819-0081-411c-87d4-8a703c754708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150e73ff-62hcr" Oct 31 19:33:03.522374 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.521860 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hjlm\" (UniqueName: \"kubernetes.io/projected/8a49b819-0081-411c-87d4-8a703c754708-kube-api-access-9hjlm\") pod \"maas-api-revoke-sub-150e73ff-62hcr\" (UID: \"8a49b819-0081-411c-87d4-8a703c754708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-150e73ff-62hcr" Oct 31 19:33:03.587617 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.587589 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl"] Oct 31 19:33:03.587962 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.587888 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:33:03.588045 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.588000 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:33:03.588045 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.588031 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:33:03.613127 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.608742 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.613127 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.608854 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.613127 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.608972 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.613127 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.609182 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9561493-df06-4f24-878e-f5d1255ac41b-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.613127 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.609286 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9561493-df06-4f24-878e-f5d1255ac41b-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.613127 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.609476 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.613127 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.609541 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9561493-df06-4f24-878e-f5d1255ac41b-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.613127 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.609579 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.613127 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.609668 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs587\" (UniqueName: \"kubernetes.io/projected/f9561493-df06-4f24-878e-f5d1255ac41b-kube-api-access-hs587\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.624535 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.624505 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150e73ff-62hcr" Oct 31 19:33:03.712591 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.711228 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs587\" (UniqueName: \"kubernetes.io/projected/f9561493-df06-4f24-878e-f5d1255ac41b-kube-api-access-hs587\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.712591 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.711309 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.712591 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.711399 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.712591 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.711435 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.712591 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.711548 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9561493-df06-4f24-878e-f5d1255ac41b-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.712591 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.711605 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9561493-df06-4f24-878e-f5d1255ac41b-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.712591 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.711676 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.712591 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.711715 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9561493-df06-4f24-878e-f5d1255ac41b-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.712591 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.711751 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.712591 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.712273 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.713325 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.712808 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.713325 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.713300 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.713873 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.713847 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.714457 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.714435 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9561493-df06-4f24-878e-f5d1255ac41b-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.726905 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.726881 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs587\" (UniqueName: \"kubernetes.io/projected/f9561493-df06-4f24-878e-f5d1255ac41b-kube-api-access-hs587\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.729702 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.729657 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9561493-df06-4f24-878e-f5d1255ac41b-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.768824 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.768718 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.770683 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.770658 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9561493-df06-4f24-878e-f5d1255ac41b-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.873739 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.873702 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:03.911965 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.911300 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:33:03.936268 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.935884 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:33:03.951250 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:03.949535 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-150e73ff-62hcr"] Oct 31 19:33:04.101388 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:04.101342 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" event={"ID":"2e946922-c762-4b9d-93af-73ab4af9fd50","Type":"ContainerStarted","Data":"01dd692f3d43a76e19c9fc79a14e16ffeee83aad69aa782fc081e5f776c7eb3d"} Oct 31 19:33:04.101607 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:04.101446 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" event={"ID":"2e946922-c762-4b9d-93af-73ab4af9fd50","Type":"ContainerStarted","Data":"921a50b1f1216615809d4ab491a217117623203af1bd024d982a8265898fc63c"} Oct 31 19:33:04.103162 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:04.103127 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150e73ff-62hcr" event={"ID":"8a49b819-0081-411c-87d4-8a703c754708","Type":"ContainerStarted","Data":"37d540b7f07a37e9f0746d23cf6758d64bae9b180ed810bf256194068a329e1c"} Oct 31 19:33:04.114854 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:04.114832 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:33:04.126338 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:04.125423 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" podStartSLOduration=2.125406061 podStartE2EDuration="2.125406061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:04.124962419 +0000 UTC m=+818.487782796" watchObservedRunningTime="2026-10-31 19:33:04.125406061 +0000 UTC m=+818.488226434" Oct 31 19:33:04.210346 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:33:04.210310 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9561493_df06_4f24_878e_f5d1255ac41b.slice/crio-3bda33a7e6f859544934a2016bcb87a2b66fb11f833cff07df2ef8d19554dcdd WatchSource:0}: Error finding container 3bda33a7e6f859544934a2016bcb87a2b66fb11f833cff07df2ef8d19554dcdd: Status 404 returned error can't find the container with id 3bda33a7e6f859544934a2016bcb87a2b66fb11f833cff07df2ef8d19554dcdd Oct 31 19:33:04.210658 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:04.210633 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr"] Oct 31 19:33:04.213408 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:04.213372 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:33:04.213493 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:04.213451 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:33:04.213535 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:04.213493 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:33:04.267489 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:04.256536 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:04.267489 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:04.266704 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 19:33:04.267489 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:04.266761 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" podUID="2e946922-c762-4b9d-93af-73ab4af9fd50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 19:33:05.115113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:05.115078 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" event={"ID":"f9561493-df06-4f24-878e-f5d1255ac41b","Type":"ContainerStarted","Data":"9d1c823c6925f1ab683f3f5a436323f906f59f17d4115922a296ef3de285a0e0"} Oct 31 19:33:05.115605 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:05.115146 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" event={"ID":"f9561493-df06-4f24-878e-f5d1255ac41b","Type":"ContainerStarted","Data":"3bda33a7e6f859544934a2016bcb87a2b66fb11f833cff07df2ef8d19554dcdd"} Oct 31 19:33:05.116984 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:05.116954 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150e73ff-62hcr" event={"ID":"8a49b819-0081-411c-87d4-8a703c754708","Type":"ContainerStarted","Data":"06280f91b511ede7a2aa6ee10c435d508f0a262c36c757001eb03460e7524e7e"} Oct 31 19:33:05.176942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:05.175747 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" podStartSLOduration=2.175728178 podStartE2EDuration="2.175728178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:05.142347634 +0000 UTC m=+819.505168008" watchObservedRunningTime="2026-10-31 19:33:05.175728178 +0000 UTC m=+819.538548550" Oct 31 19:33:05.178284 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:05.178243 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150e73ff-62hcr" podStartSLOduration=2.178233971 podStartE2EDuration="2.178233971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:05.165695925 +0000 UTC m=+819.528516336" watchObservedRunningTime="2026-10-31 19:33:05.178233971 +0000 UTC m=+819.541054343" Oct 31 19:33:05.253933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:05.253333 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 19:33:05.253933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:05.253397 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" podUID="2e946922-c762-4b9d-93af-73ab4af9fd50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 19:33:05.877935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:05.877714 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:05.877935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:05.877844 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 19:33:05.877935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:05.877887 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" podUID="f9561493-df06-4f24-878e-f5d1255ac41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 19:33:06.182993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:06.171413 4910 generic.go:356] "Generic (PLEG): container finished" podID="8a49b819-0081-411c-87d4-8a703c754708" containerID="06280f91b511ede7a2aa6ee10c435d508f0a262c36c757001eb03460e7524e7e" exitCode=0 Oct 31 19:33:06.182993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:06.172984 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150e73ff-62hcr" event={"ID":"8a49b819-0081-411c-87d4-8a703c754708","Type":"ContainerDied","Data":"06280f91b511ede7a2aa6ee10c435d508f0a262c36c757001eb03460e7524e7e"} Oct 31 19:33:06.270933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:06.270334 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 19:33:06.270933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:06.270397 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" podUID="2e946922-c762-4b9d-93af-73ab4af9fd50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 19:33:06.876934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:06.876383 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 19:33:06.876934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:06.876448 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" podUID="f9561493-df06-4f24-878e-f5d1255ac41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 19:33:07.257938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:07.257328 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 19:33:07.257938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:07.257387 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" podUID="2e946922-c762-4b9d-93af-73ab4af9fd50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 19:33:07.817794 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:07.817282 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150e73ff-62hcr" Oct 31 19:33:07.854376 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:07.854338 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a49b819-0081-411c-87d4-8a703c754708-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a49b819-0081-411c-87d4-8a703c754708-maas-api-xks-ca\") pod \"8a49b819-0081-411c-87d4-8a703c754708\" (UID: \"8a49b819-0081-411c-87d4-8a703c754708\") " Oct 31 19:33:07.854645 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:07.854625 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a49b819-0081-411c-87d4-8a703c754708-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a49b819-0081-411c-87d4-8a703c754708-maas-api-openshift-ca\") pod \"8a49b819-0081-411c-87d4-8a703c754708\" (UID: \"8a49b819-0081-411c-87d4-8a703c754708\") " Oct 31 19:33:07.870034 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:07.870005 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a49b819-0081-411c-87d4-8a703c754708-kube-api-access-9hjlm\" (UniqueName: \"kubernetes.io/projected/8a49b819-0081-411c-87d4-8a703c754708-kube-api-access-9hjlm\") pod \"8a49b819-0081-411c-87d4-8a703c754708\" (UID: \"8a49b819-0081-411c-87d4-8a703c754708\") " Oct 31 19:33:07.871364 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:07.871336 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a49b819-0081-411c-87d4-8a703c754708-maas-api-openshift-ca" pod "8a49b819-0081-411c-87d4-8a703c754708" (UID: "8a49b819-0081-411c-87d4-8a703c754708"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:33:07.876933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:07.874313 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 19:33:07.876933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:07.874366 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" podUID="f9561493-df06-4f24-878e-f5d1255ac41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 19:33:07.904342 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:07.904304 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a49b819-0081-411c-87d4-8a703c754708-maas-api-xks-ca" pod "8a49b819-0081-411c-87d4-8a703c754708" (UID: "8a49b819-0081-411c-87d4-8a703c754708"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:33:07.912184 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:07.912127 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a49b819-0081-411c-87d4-8a703c754708-kube-api-access-9hjlm" pod "8a49b819-0081-411c-87d4-8a703c754708" (UID: "8a49b819-0081-411c-87d4-8a703c754708"). InnerVolumeSpecName "kube-api-access-9hjlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:33:07.972800 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:07.972713 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a49b819-0081-411c-87d4-8a703c754708-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:33:07.972800 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:07.972757 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a49b819-0081-411c-87d4-8a703c754708-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:33:07.972800 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:07.972773 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hjlm\" (UniqueName: \"kubernetes.io/projected/8a49b819-0081-411c-87d4-8a703c754708-kube-api-access-9hjlm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:33:08.213243 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:08.213208 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150e73ff-62hcr" event={"ID":"8a49b819-0081-411c-87d4-8a703c754708","Type":"ContainerDied","Data":"37d540b7f07a37e9f0746d23cf6758d64bae9b180ed810bf256194068a329e1c"} Oct 31 19:33:08.213415 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:08.213258 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37d540b7f07a37e9f0746d23cf6758d64bae9b180ed810bf256194068a329e1c" Oct 31 19:33:08.213415 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:08.213311 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-150e73ff-62hcr" Oct 31 19:33:08.266936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:08.256322 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 19:33:08.266936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:08.256375 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" podUID="2e946922-c762-4b9d-93af-73ab4af9fd50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 19:33:08.879527 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:08.879490 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 19:33:08.879801 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:08.879771 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" podUID="f9561493-df06-4f24-878e-f5d1255ac41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 19:33:09.280950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:09.280569 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 19:33:09.280950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:09.280646 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" podUID="2e946922-c762-4b9d-93af-73ab4af9fd50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 19:33:09.876335 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:09.876299 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 19:33:09.876581 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:09.876554 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" podUID="f9561493-df06-4f24-878e-f5d1255ac41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 19:33:10.257202 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:10.257109 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 31 19:33:10.257465 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:10.257431 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" podUID="2e946922-c762-4b9d-93af-73ab4af9fd50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 31 19:33:10.874545 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:10.874506 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 19:33:10.875471 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:10.875083 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" podUID="f9561493-df06-4f24-878e-f5d1255ac41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 19:33:11.655414 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:11.655378 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:11.656331 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:11.656307 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:11.659085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:11.659061 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:33:12.292152 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:12.292114 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:12.293198 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:12.293172 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:12.294011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:12.293988 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:33:12.715849 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:33:12.715810 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:33:14.808935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:14.808469 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:14.812930 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:14.811986 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:14.965093 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:14.965043 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:15.032054 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:15.032005 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:15.097327 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:15.097223 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded" Oct 31 19:33:15.187111 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:15.187042 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:15.188287 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:15.188241 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:33:15.229652 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:15.229075 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:15.339024 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:15.329214 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:15.458186 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:15.456999 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:15.481940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:15.480014 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:15.482569 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:15.482538 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:15.606402 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:15.606354 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:15.814539 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:15.814112 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:15.834573 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:15.834538 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:33:15.951480 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:15.951433 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:16.151999 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:16.088677 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:16.160933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:16.155476 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:33:16.211491 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:16.211246 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:16.232354 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:16.218979 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:33:16.232354 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:16.219609 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:33:16.232354 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:16.219668 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:33:16.253975 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:16.247391 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:16.331753 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:16.254227 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:33:16.337952 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:16.332187 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:16.346658 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:16.346625 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:33:16.477903 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:16.477806 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:33:16.535935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:16.479780 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:33:16.702944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:16.654762 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:16.706935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:16.656610 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": context deadline exceeded" Oct 31 19:33:16.779118 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:16.679397 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:33:16.780687 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:16.726543 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:33:16.788935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:16.780992 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:33:16.843936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:16.818539 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:33:16.881968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:16.845678 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:16.962054 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:16.962008 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:16.987937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:16.962831 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:33:17.156935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:17.114770 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:17.216937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:17.191542 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:17.278936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:17.217258 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:17.278936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:17.229300 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:17.296968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:17.247778 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:33:17.319153 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:17.264475 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:17.366938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:17.334540 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:17.508993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:17.502733 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:33:17.508993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:17.504482 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:33:17.664272 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:17.664220 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:17.703813 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:17.703770 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:33:17.845466 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:17.845416 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:17.889937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:17.882057 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:33:17.889937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:17.883137 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:18.125995 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:18.098745 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:33:18.125995 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:18.103223 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:33:18.127301 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:18.127259 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:18.302451 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:18.302403 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:18.304185 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:18.304059 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:18.386831 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:18.386721 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:18.423058 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:18.423009 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:18.537554 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:18.537500 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:18.538261 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:18.538226 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": context deadline exceeded" Oct 31 19:33:18.566205 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:18.566159 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:18.586938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:18.567425 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:33:18.704613 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:18.704508 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:18.738419 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:18.738360 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:18.781939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:18.746110 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:18.782256 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:18.750349 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:33:18.908063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:18.908020 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:33:18.908855 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:18.907855 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:33:19.074994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:19.072751 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:19.107935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:19.106999 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:19.227935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:19.216697 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 19:33:19.381991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:19.348127 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:19.381991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:19.375137 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:33:19.408157 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:19.404182 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:19.439935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:19.431221 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:19.540687 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:19.540645 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:19.569233 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:19.569187 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:19.705258 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:19.705151 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:19.750947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:19.750021 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:20.100063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:20.100016 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:33:20.116063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:20.116026 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:33:20.164949 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:20.125979 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:33:20.164949 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:20.134514 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:33:20.252937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:20.189288 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:33:20.252937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:20.232106 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:20.273983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:20.273495 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:33:20.278948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:20.274526 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:33:20.352995 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:20.348444 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:20.510996 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:20.412001 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:20.510996 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:20.468230 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:33:20.510996 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:20.470277 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:20.530000 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:20.529566 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:20.535582 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:20.535541 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:33:20.550950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:20.541257 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:20.557316 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:20.557275 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:33:20.572092 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:20.570466 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:20.788020 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:20.787925 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:33:21.015469 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:21.015437 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:33:21.056056 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:21.043835 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:33:21.124766 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:21.124736 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:33:21.149352 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:21.149170 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:33:21.149948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:21.149905 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:33:21.155772 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:21.155750 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:33:21.156580 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:21.134768 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:21.176510 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:21.176484 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:33:21.197534 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:21.197506 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:33:24.643479 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:24.643442 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60e0aed9-5j9f4"] Oct 31 19:33:24.644096 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:24.644081 4910 cpu_manager.go:414] "containerMap: removing container" podUID="8a49b819-0081-411c-87d4-8a703c754708" containerName="revoke-keys" Oct 31 19:33:24.644096 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:24.644096 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a49b819-0081-411c-87d4-8a703c754708" containerName="revoke-keys" Oct 31 19:33:24.644238 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:24.644229 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a49b819-0081-411c-87d4-8a703c754708" containerName="revoke-keys" Oct 31 19:33:24.649614 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:24.649552 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e0aed9-5j9f4" Oct 31 19:33:24.656440 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:24.653735 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:33:24.656440 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:24.655243 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:33:24.659506 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:24.659481 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60e0aed9-5j9f4"] Oct 31 19:33:24.782311 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:24.782271 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eed7b17-62bf-41b8-90bf-95c5ec361bc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60e0aed9-5j9f4\" (UID: \"8eed7b17-62bf-41b8-90bf-95c5ec361bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e0aed9-5j9f4" Oct 31 19:33:24.782543 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:24.782512 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt89t\" (UniqueName: \"kubernetes.io/projected/8eed7b17-62bf-41b8-90bf-95c5ec361bc3-kube-api-access-nt89t\") pod \"maas-api-revoke-sub-60e0aed9-5j9f4\" (UID: \"8eed7b17-62bf-41b8-90bf-95c5ec361bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e0aed9-5j9f4" Oct 31 19:33:24.782753 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:24.782736 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eed7b17-62bf-41b8-90bf-95c5ec361bc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60e0aed9-5j9f4\" (UID: \"8eed7b17-62bf-41b8-90bf-95c5ec361bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e0aed9-5j9f4" Oct 31 19:33:24.889321 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:24.889090 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eed7b17-62bf-41b8-90bf-95c5ec361bc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60e0aed9-5j9f4\" (UID: \"8eed7b17-62bf-41b8-90bf-95c5ec361bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e0aed9-5j9f4" Oct 31 19:33:24.889321 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:24.889273 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt89t\" (UniqueName: \"kubernetes.io/projected/8eed7b17-62bf-41b8-90bf-95c5ec361bc3-kube-api-access-nt89t\") pod \"maas-api-revoke-sub-60e0aed9-5j9f4\" (UID: \"8eed7b17-62bf-41b8-90bf-95c5ec361bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e0aed9-5j9f4" Oct 31 19:33:24.889605 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:24.889530 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eed7b17-62bf-41b8-90bf-95c5ec361bc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60e0aed9-5j9f4\" (UID: \"8eed7b17-62bf-41b8-90bf-95c5ec361bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e0aed9-5j9f4" Oct 31 19:33:24.902001 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:24.901156 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eed7b17-62bf-41b8-90bf-95c5ec361bc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60e0aed9-5j9f4\" (UID: \"8eed7b17-62bf-41b8-90bf-95c5ec361bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e0aed9-5j9f4" Oct 31 19:33:24.907161 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:24.907128 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eed7b17-62bf-41b8-90bf-95c5ec361bc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60e0aed9-5j9f4\" (UID: \"8eed7b17-62bf-41b8-90bf-95c5ec361bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e0aed9-5j9f4" Oct 31 19:33:24.914609 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:24.914575 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt89t\" (UniqueName: \"kubernetes.io/projected/8eed7b17-62bf-41b8-90bf-95c5ec361bc3-kube-api-access-nt89t\") pod \"maas-api-revoke-sub-60e0aed9-5j9f4\" (UID: \"8eed7b17-62bf-41b8-90bf-95c5ec361bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e0aed9-5j9f4" Oct 31 19:33:24.970608 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:24.970572 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e0aed9-5j9f4" Oct 31 19:33:25.280504 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:33:25.280461 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eed7b17_62bf_41b8_90bf_95c5ec361bc3.slice/crio-70a361521572a39d70e6983de9d51d21cf3d058f6f3f22ff3ff6a2532d713ca7 WatchSource:0}: Error finding container 70a361521572a39d70e6983de9d51d21cf3d058f6f3f22ff3ff6a2532d713ca7: Status 404 returned error can't find the container with id 70a361521572a39d70e6983de9d51d21cf3d058f6f3f22ff3ff6a2532d713ca7 Oct 31 19:33:25.283372 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:25.283344 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60e0aed9-5j9f4"] Oct 31 19:33:26.079276 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:26.079231 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e0aed9-5j9f4" event={"ID":"8eed7b17-62bf-41b8-90bf-95c5ec361bc3","Type":"ContainerStarted","Data":"63064eddc5527ba3eda0deaaf2e3e5f6f678588a2f33e0f608fd7e03da58147f"} Oct 31 19:33:26.079659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:26.079304 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e0aed9-5j9f4" event={"ID":"8eed7b17-62bf-41b8-90bf-95c5ec361bc3","Type":"ContainerStarted","Data":"70a361521572a39d70e6983de9d51d21cf3d058f6f3f22ff3ff6a2532d713ca7"} Oct 31 19:33:26.109273 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:26.109218 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e0aed9-5j9f4" podStartSLOduration=2.1092051019999998 podStartE2EDuration="2.109205102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:26.106099151 +0000 UTC m=+840.468919522" watchObservedRunningTime="2026-10-31 19:33:26.109205102 +0000 UTC m=+840.472025474" Oct 31 19:33:27.098035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:27.097996 4910 generic.go:356] "Generic (PLEG): container finished" podID="8eed7b17-62bf-41b8-90bf-95c5ec361bc3" containerID="63064eddc5527ba3eda0deaaf2e3e5f6f678588a2f33e0f608fd7e03da58147f" exitCode=0 Oct 31 19:33:27.098523 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:27.098061 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e0aed9-5j9f4" event={"ID":"8eed7b17-62bf-41b8-90bf-95c5ec361bc3","Type":"ContainerDied","Data":"63064eddc5527ba3eda0deaaf2e3e5f6f678588a2f33e0f608fd7e03da58147f"} Oct 31 19:33:28.371101 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:28.371035 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e0aed9-5j9f4" Oct 31 19:33:28.542314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:28.542279 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eed7b17-62bf-41b8-90bf-95c5ec361bc3-kube-api-access-nt89t\" (UniqueName: \"kubernetes.io/projected/8eed7b17-62bf-41b8-90bf-95c5ec361bc3-kube-api-access-nt89t\") pod \"8eed7b17-62bf-41b8-90bf-95c5ec361bc3\" (UID: \"8eed7b17-62bf-41b8-90bf-95c5ec361bc3\") " Oct 31 19:33:28.542507 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:28.542407 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eed7b17-62bf-41b8-90bf-95c5ec361bc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eed7b17-62bf-41b8-90bf-95c5ec361bc3-maas-api-openshift-ca\") pod \"8eed7b17-62bf-41b8-90bf-95c5ec361bc3\" (UID: \"8eed7b17-62bf-41b8-90bf-95c5ec361bc3\") " Oct 31 19:33:28.542573 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:28.542559 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eed7b17-62bf-41b8-90bf-95c5ec361bc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eed7b17-62bf-41b8-90bf-95c5ec361bc3-maas-api-xks-ca\") pod \"8eed7b17-62bf-41b8-90bf-95c5ec361bc3\" (UID: \"8eed7b17-62bf-41b8-90bf-95c5ec361bc3\") " Oct 31 19:33:28.542856 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:28.542776 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eed7b17-62bf-41b8-90bf-95c5ec361bc3-maas-api-openshift-ca" pod "8eed7b17-62bf-41b8-90bf-95c5ec361bc3" (UID: "8eed7b17-62bf-41b8-90bf-95c5ec361bc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:33:28.543175 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:28.543157 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eed7b17-62bf-41b8-90bf-95c5ec361bc3-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:33:28.545038 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:28.545011 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eed7b17-62bf-41b8-90bf-95c5ec361bc3-maas-api-xks-ca" pod "8eed7b17-62bf-41b8-90bf-95c5ec361bc3" (UID: "8eed7b17-62bf-41b8-90bf-95c5ec361bc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:33:28.545237 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:28.545213 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eed7b17-62bf-41b8-90bf-95c5ec361bc3-kube-api-access-nt89t" pod "8eed7b17-62bf-41b8-90bf-95c5ec361bc3" (UID: "8eed7b17-62bf-41b8-90bf-95c5ec361bc3"). InnerVolumeSpecName "kube-api-access-nt89t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:33:28.645074 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:28.644985 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eed7b17-62bf-41b8-90bf-95c5ec361bc3-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:33:28.645074 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:28.645017 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt89t\" (UniqueName: \"kubernetes.io/projected/8eed7b17-62bf-41b8-90bf-95c5ec361bc3-kube-api-access-nt89t\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:33:29.121852 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:29.121811 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e0aed9-5j9f4" Oct 31 19:33:29.122054 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:29.121874 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e0aed9-5j9f4" event={"ID":"8eed7b17-62bf-41b8-90bf-95c5ec361bc3","Type":"ContainerDied","Data":"70a361521572a39d70e6983de9d51d21cf3d058f6f3f22ff3ff6a2532d713ca7"} Oct 31 19:33:29.122054 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:29.121948 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70a361521572a39d70e6983de9d51d21cf3d058f6f3f22ff3ff6a2532d713ca7" Oct 31 19:33:29.176232 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:33:29.176197 4910 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eed7b17_62bf_41b8_90bf_95c5ec361bc3.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eed7b17_62bf_41b8_90bf_95c5ec361bc3.slice/crio-70a361521572a39d70e6983de9d51d21cf3d058f6f3f22ff3ff6a2532d713ca7\": RecentStats: unable to find data in memory cache]" Oct 31 19:33:31.841467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:31.841429 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58c55970-lql9j"] Oct 31 19:33:31.842455 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:31.842432 4910 cpu_manager.go:414] "containerMap: removing container" podUID="8eed7b17-62bf-41b8-90bf-95c5ec361bc3" containerName="revoke-keys" Oct 31 19:33:31.842518 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:31.842460 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eed7b17-62bf-41b8-90bf-95c5ec361bc3" containerName="revoke-keys" Oct 31 19:33:31.842705 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:31.842692 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eed7b17-62bf-41b8-90bf-95c5ec361bc3" containerName="revoke-keys" Oct 31 19:33:31.847290 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:31.847266 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c55970-lql9j" Oct 31 19:33:31.854048 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:31.854013 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58c55970-lql9j"] Oct 31 19:33:31.856043 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:31.856012 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:33:31.856188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:31.856078 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:33:31.900728 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:31.900694 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a666bdeb-b0ad-450e-b44a-a05ab6835ace-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58c55970-lql9j\" (UID: \"a666bdeb-b0ad-450e-b44a-a05ab6835ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c55970-lql9j" Oct 31 19:33:31.900963 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:31.900902 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a666bdeb-b0ad-450e-b44a-a05ab6835ace-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58c55970-lql9j\" (UID: \"a666bdeb-b0ad-450e-b44a-a05ab6835ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c55970-lql9j" Oct 31 19:33:31.901137 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:31.901112 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr2jf\" (UniqueName: \"kubernetes.io/projected/a666bdeb-b0ad-450e-b44a-a05ab6835ace-kube-api-access-pr2jf\") pod \"maas-api-revoke-sub-58c55970-lql9j\" (UID: \"a666bdeb-b0ad-450e-b44a-a05ab6835ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c55970-lql9j" Oct 31 19:33:32.001936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:32.001892 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr2jf\" (UniqueName: \"kubernetes.io/projected/a666bdeb-b0ad-450e-b44a-a05ab6835ace-kube-api-access-pr2jf\") pod \"maas-api-revoke-sub-58c55970-lql9j\" (UID: \"a666bdeb-b0ad-450e-b44a-a05ab6835ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c55970-lql9j" Oct 31 19:33:32.002106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:32.002003 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a666bdeb-b0ad-450e-b44a-a05ab6835ace-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58c55970-lql9j\" (UID: \"a666bdeb-b0ad-450e-b44a-a05ab6835ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c55970-lql9j" Oct 31 19:33:32.002178 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:32.002161 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a666bdeb-b0ad-450e-b44a-a05ab6835ace-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58c55970-lql9j\" (UID: \"a666bdeb-b0ad-450e-b44a-a05ab6835ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c55970-lql9j" Oct 31 19:33:32.002806 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:32.002779 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a666bdeb-b0ad-450e-b44a-a05ab6835ace-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58c55970-lql9j\" (UID: \"a666bdeb-b0ad-450e-b44a-a05ab6835ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c55970-lql9j" Oct 31 19:33:32.004400 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:32.004376 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a666bdeb-b0ad-450e-b44a-a05ab6835ace-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58c55970-lql9j\" (UID: \"a666bdeb-b0ad-450e-b44a-a05ab6835ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c55970-lql9j" Oct 31 19:33:32.014584 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:32.014562 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr2jf\" (UniqueName: \"kubernetes.io/projected/a666bdeb-b0ad-450e-b44a-a05ab6835ace-kube-api-access-pr2jf\") pod \"maas-api-revoke-sub-58c55970-lql9j\" (UID: \"a666bdeb-b0ad-450e-b44a-a05ab6835ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c55970-lql9j" Oct 31 19:33:32.205819 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:32.205739 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c55970-lql9j" Oct 31 19:33:32.500366 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:33:32.500334 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda666bdeb_b0ad_450e_b44a_a05ab6835ace.slice/crio-32d599252c9e7b03acbcca8418691d99041a1879ee5a3d03139a54e094b8a125 WatchSource:0}: Error finding container 32d599252c9e7b03acbcca8418691d99041a1879ee5a3d03139a54e094b8a125: Status 404 returned error can't find the container with id 32d599252c9e7b03acbcca8418691d99041a1879ee5a3d03139a54e094b8a125 Oct 31 19:33:32.502055 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:32.502029 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58c55970-lql9j"] Oct 31 19:33:33.161999 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.161870 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c55970-lql9j" event={"ID":"a666bdeb-b0ad-450e-b44a-a05ab6835ace","Type":"ContainerStarted","Data":"d1bb99c3b798c0c0ab9d27b687407b50c7734a6f507f1c4266e8330a1766c6e3"} Oct 31 19:33:33.161999 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.161947 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c55970-lql9j" event={"ID":"a666bdeb-b0ad-450e-b44a-a05ab6835ace","Type":"ContainerStarted","Data":"32d599252c9e7b03acbcca8418691d99041a1879ee5a3d03139a54e094b8a125"} Oct 31 19:33:33.182382 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.182350 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w"] Oct 31 19:33:33.186861 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.186839 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.204031 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.203457 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c55970-lql9j" podStartSLOduration=2.203440094 podStartE2EDuration="2.203440094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:33.197509601 +0000 UTC m=+847.560330000" watchObservedRunningTime="2026-10-31 19:33:33.203440094 +0000 UTC m=+847.566260467" Oct 31 19:33:33.216284 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.216250 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w"] Oct 31 19:33:33.224171 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.219741 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.224171 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.219841 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.224171 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.219901 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.224171 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.220008 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be3bf6bf-4863-4489-b924-3de83b589c8e-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.224171 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.220142 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.224171 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.220304 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.224171 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.220396 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw28n\" (UniqueName: \"kubernetes.io/projected/be3bf6bf-4863-4489-b924-3de83b589c8e-kube-api-access-pw28n\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.224171 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.220469 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.224171 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.220515 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.322459 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.322425 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be3bf6bf-4863-4489-b924-3de83b589c8e-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.322459 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.322472 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.322724 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.322514 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.322724 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.322567 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw28n\" (UniqueName: \"kubernetes.io/projected/be3bf6bf-4863-4489-b924-3de83b589c8e-kube-api-access-pw28n\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.322724 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.322626 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.322724 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.322697 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.322942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.322841 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.323007 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.322952 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.323063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.323015 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.323138 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.323112 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.323448 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.323263 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be3bf6bf-4863-4489-b924-3de83b589c8e-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.323448 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.323390 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.323596 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.323454 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.323596 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.323476 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.325083 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.325057 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.325668 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.325644 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.378005 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.377972 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.378150 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.378027 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw28n\" (UniqueName: \"kubernetes.io/projected/be3bf6bf-4863-4489-b924-3de83b589c8e-kube-api-access-pw28n\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.468113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.468034 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg"] Oct 31 19:33:33.473684 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.473663 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.497970 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.497942 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg"] Oct 31 19:33:33.519948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.514084 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:33.530135 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.528220 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj"] Oct 31 19:33:33.530135 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.529877 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.530135 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.529938 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.530135 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.530004 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r8hc\" (UniqueName: \"kubernetes.io/projected/71387fb8-12d0-4620-a56c-7b4cd6ba554e-kube-api-access-8r8hc\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.530135 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.530072 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.530135 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.530139 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.530467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.530190 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.530467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.530223 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.530467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.530360 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.530467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.530431 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.536164 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.533949 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.546676 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.546647 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj"] Oct 31 19:33:33.634240 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.632311 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.634384 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.634288 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.634384 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.634334 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.634384 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.634362 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.634539 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.634392 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.634539 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.634487 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.637255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.635206 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.637255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.635630 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.637255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.635306 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.637255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.635760 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.637255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.635842 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.637255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.636063 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.637255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.636096 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.637255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.636141 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.637255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.636170 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.637255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.636221 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvrv2\" (UniqueName: \"kubernetes.io/projected/1540ff89-ac68-464d-b39f-c01ef4ebd62c-kube-api-access-bvrv2\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.637255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.636269 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r8hc\" (UniqueName: \"kubernetes.io/projected/71387fb8-12d0-4620-a56c-7b4cd6ba554e-kube-api-access-8r8hc\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.637255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.636355 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.637255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.636412 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.637255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.636435 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.637255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.636467 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.637255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.636695 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.637255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.636820 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.648599 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.648552 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.650502 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.650458 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r8hc\" (UniqueName: \"kubernetes.io/projected/71387fb8-12d0-4620-a56c-7b4cd6ba554e-kube-api-access-8r8hc\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.652906 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.652861 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.654468 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.654447 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.795223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.742014 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.795223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.742092 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.795223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.742133 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.795223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.742224 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.795223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.742272 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.795223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.742344 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.795223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.742477 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.795223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.742513 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.795223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.742559 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvrv2\" (UniqueName: \"kubernetes.io/projected/1540ff89-ac68-464d-b39f-c01ef4ebd62c-kube-api-access-bvrv2\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.795223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.743430 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.795223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.768301 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.795223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.768875 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.795223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.769142 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.795223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.776568 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvrv2\" (UniqueName: \"kubernetes.io/projected/1540ff89-ac68-464d-b39f-c01ef4ebd62c-kube-api-access-bvrv2\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.795223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.777123 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.795223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.784885 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.795223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.788627 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.795223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.790009 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:33.799531 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.799476 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.868561 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.868043 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w"] Oct 31 19:33:33.877159 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.877128 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:33.951985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.945303 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:33:33.951985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.945398 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:33:33.951985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:33.945453 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:33:34.865745 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:34.865684 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:33:34.866353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:34.866328 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:33:34.869056 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:34.869008 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:33:34.893166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:34.892043 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" event={"ID":"be3bf6bf-4863-4489-b924-3de83b589c8e","Type":"ContainerStarted","Data":"ad552c55fefc09844acf4440ad2a912915e6c083570317a80245046124cd44df"} Oct 31 19:33:34.922960 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:34.899854 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt"] Oct 31 19:33:34.943040 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:34.929968 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:34.987938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:34.973567 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt"] Oct 31 19:33:35.085525 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.085484 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.085975 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.085942 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.086218 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.086200 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.086365 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.086347 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwzvc\" (UniqueName: \"kubernetes.io/projected/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-kube-api-access-kwzvc\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.086542 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.086523 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.086683 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.086665 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.086785 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.086771 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.086887 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.086875 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.087125 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.087107 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.087561 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.087514 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:35.134220 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.117356 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:35.205226 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.202270 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.205226 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.202425 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.205226 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.202502 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.205226 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.202641 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.205226 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.202698 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwzvc\" (UniqueName: \"kubernetes.io/projected/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-kube-api-access-kwzvc\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.205226 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.202796 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.205226 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.202845 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.205226 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.202872 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.205226 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.202967 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.301936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.236596 4910 generic.go:356] "Generic (PLEG): container finished" podID="a666bdeb-b0ad-450e-b44a-a05ab6835ace" containerID="d1bb99c3b798c0c0ab9d27b687407b50c7734a6f507f1c4266e8330a1766c6e3" exitCode=0 Oct 31 19:33:35.301936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.236674 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c55970-lql9j" event={"ID":"a666bdeb-b0ad-450e-b44a-a05ab6835ace","Type":"ContainerDied","Data":"d1bb99c3b798c0c0ab9d27b687407b50c7734a6f507f1c4266e8330a1766c6e3"} Oct 31 19:33:35.308063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.308013 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:35.308738 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.308711 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:35.317658 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.317554 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:35.338323 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.320613 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:35.359239 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.358715 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.359239 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.359107 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.362471 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.362188 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.362471 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.362427 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.363010 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.362979 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.376031 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.375980 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:35.466931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.445993 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg"] Oct 31 19:33:35.478018 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.477948 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwzvc\" (UniqueName: \"kubernetes.io/projected/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-kube-api-access-kwzvc\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.561840 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.561791 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.566461 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.566435 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.643075 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.639275 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.678748 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.677538 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:35.678748 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.678454 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:33:35.772180 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.748196 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:35.929825 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.903632 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:35.931977 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.931257 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:35.994181 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:35.994089 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:33:36.055887 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:36.052259 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:36.219412 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:36.210346 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:36.219412 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:36.210681 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:33:36.225313 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:36.222342 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:33:36.225313 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:36.222398 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:33:36.359951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:36.350193 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:36.359951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:36.351488 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:36.359951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:36.351785 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:33:36.359951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:36.351888 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:33:36.359951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:36.351963 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:33:36.359951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:36.352005 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:33:36.376981 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:36.375589 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:36.376981 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:36.376076 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:33:36.401949 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:36.377814 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:36.401949 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:36.378204 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:33:36.723406 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:36.682995 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:36.723406 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:36.683294 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:33:36.723406 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:36.683541 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:33:36.723406 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:36.683599 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:33:36.922938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:36.906162 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:36.922938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:36.914053 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:33:36.951606 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:36.951559 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:36.990777 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:36.990689 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:33:36.994311 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:36.994261 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:33:37.003940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:36.995514 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:33:37.005243 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:37.005210 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" event={"ID":"be3bf6bf-4863-4489-b924-3de83b589c8e","Type":"ContainerStarted","Data":"eb4d5dc8612c55538b4fe5a87db69d023da6d807ad5f4617dbbf7d0858497bb3"} Oct 31 19:33:37.133945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:37.128677 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:33:37.133945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:37.129780 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:33:37.235956 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:37.232164 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:33:37.237127 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:37.237086 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:33:37.377968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:37.275693 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:37.400094 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:37.310220 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded" Oct 31 19:33:37.400406 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:37.376191 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:37.426315 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:37.376452 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:37.426315 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:37.376485 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:37.426315 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:37.376558 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:37.426315 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:37.399931 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:37.469937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:37.453087 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:33:37.511191 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:37.511159 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:33:37.516934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:37.516759 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:37.516934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:37.516884 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 31 19:33:37.517184 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:37.516949 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" podUID="be3bf6bf-4863-4489-b924-3de83b589c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 31 19:33:37.695129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:37.695071 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:38.007983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:38.004029 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:33:38.070439 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:38.010999 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:33:38.177264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:38.147719 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:33:38.441575 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:38.330466 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:33:38.441575 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:38.330626 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:33:38.441575 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:38.184221 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:38.441575 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:38.184550 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:38.441575 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:38.238087 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:38.441575 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:38.439162 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:33:38.441575 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:38.440503 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:33:38.444257 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:38.443222 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:38.488436 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:38.453013 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:38.488436 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:38.453321 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:33:38.488436 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:38.453523 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 19:33:38.540979 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:38.533675 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:38.603201 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:38.603084 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 31 19:33:38.603468 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:38.603406 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" podUID="be3bf6bf-4863-4489-b924-3de83b589c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 31 19:33:38.747812 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:38.747762 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:38.994993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:38.966392 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" event={"ID":"71387fb8-12d0-4620-a56c-7b4cd6ba554e","Type":"ContainerStarted","Data":"73f34509f1c6127f6aef35bdae3a71f7ebc24ae228fddabd88fe063c361a9044"} Oct 31 19:33:39.079995 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:38.977659 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:33:39.204191 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:39.200192 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:33:39.204191 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:39.200634 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:33:39.277988 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:39.222780 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:33:39.287203 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:39.287138 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:33:39.291110 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:39.291053 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:33:39.338672 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:39.334045 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:39.338672 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:39.334463 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:33:39.338672 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:39.334831 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:33:39.454947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:39.454073 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:39.454947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:39.454533 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:33:39.455236 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:39.455087 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:39.459945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:39.455386 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:33:39.459945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:39.459135 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:33:39.485940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:39.484177 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:39.591253 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:39.544185 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:33:39.639534 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:39.518307 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:33:39.706310 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:39.706268 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj"] Oct 31 19:33:39.772031 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:39.771994 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 31 19:33:39.836022 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:39.835980 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" podUID="be3bf6bf-4863-4489-b924-3de83b589c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 31 19:33:40.126139 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:40.111181 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:33:40.200580 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:40.200533 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:40.219252 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:40.200851 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:33:40.585936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:40.530660 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:40.585936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:40.532747 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:33:40.585936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:40.533134 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:33:40.585936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:40.533565 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:33:40.585936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:40.534420 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:40.585936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:40.534765 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:40.585936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:40.535046 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:40.585936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:40.535335 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:40.585936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:40.535582 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:40.620932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:40.609179 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:40.680991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:40.667550 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:41.170294 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:41.170258 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 31 19:33:41.317730 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:41.317667 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" podUID="be3bf6bf-4863-4489-b924-3de83b589c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 31 19:33:41.324759 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:41.323974 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:33:41.405738 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:33:41.370508 4910 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.596s" Oct 31 19:33:41.476064 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:41.476028 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:33:41.488232 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:41.476337 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" event={"ID":"0ba0cb41-a7be-462c-9e92-7dccedd1d92f","Type":"ContainerStarted","Data":"6fe550351e235806ea6f50d2a95f4ce4a863f0528ddec2da9bdd752f6c621511"} Oct 31 19:33:41.488538 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:41.488503 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" event={"ID":"71387fb8-12d0-4620-a56c-7b4cd6ba554e","Type":"ContainerStarted","Data":"f167e66498f2b6403f6f04252b222a5721b5ad55853f3338d44400fba5644a10"} Oct 31 19:33:41.488674 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:41.405635 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:33:41.509935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:41.405700 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:41.556075 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:41.556027 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:41.558026 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:41.557991 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:41.662710 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:41.662665 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 31 19:33:41.662990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:41.662958 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" podUID="be3bf6bf-4863-4489-b924-3de83b589c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 31 19:33:41.672234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:41.672191 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 19:33:41.700163 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:41.700105 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:33:42.209350 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:41.748855 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 19:33:42.210057 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:41.748896 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:33:42.223986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:41.748933 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:33:42.246971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:41.755129 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:33:42.247261 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:41.794070 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:33:42.248741 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:41.794125 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:42.249499 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:41.794755 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:42.288963 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:41.811590 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 19:33:42.288963 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:42.281509 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" podUID="71387fb8-12d0-4620-a56c-7b4cd6ba554e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 19:33:42.288963 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:42.072169 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded" Oct 31 19:33:42.288963 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:42.072109 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:42.339548 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:42.339504 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:42.375180 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:42.375120 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded" Oct 31 19:33:42.508985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:42.503061 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:42.520634 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:42.520590 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:33:42.587139 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:42.587071 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:33:42.631033 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:42.630990 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:42.688232 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:42.688198 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 31 19:33:42.743732 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:42.743674 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" podUID="be3bf6bf-4863-4489-b924-3de83b589c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 31 19:33:42.755027 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:42.754975 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:42.853932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:42.853704 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 19:33:42.853932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:42.853783 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" podUID="71387fb8-12d0-4620-a56c-7b4cd6ba554e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 19:33:43.064938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:42.994320 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" event={"ID":"1540ff89-ac68-464d-b39f-c01ef4ebd62c","Type":"ContainerStarted","Data":"e9aa7a882a703f4275e6e94b22b64aafa941578e41e9e9dd69b21804b76c4eb5"} Oct 31 19:33:43.154267 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.154188 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c55970-lql9j" Oct 31 19:33:43.214998 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.212613 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt"] Oct 31 19:33:43.247992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.247941 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:43.269254 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.269208 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:33:43.298410 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.298143 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 19:33:43.321141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.321088 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:33:43.353086 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.352662 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a666bdeb-b0ad-450e-b44a-a05ab6835ace-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a666bdeb-b0ad-450e-b44a-a05ab6835ace-maas-api-openshift-ca\") pod \"a666bdeb-b0ad-450e-b44a-a05ab6835ace\" (UID: \"a666bdeb-b0ad-450e-b44a-a05ab6835ace\") " Oct 31 19:33:43.353086 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.352819 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a666bdeb-b0ad-450e-b44a-a05ab6835ace-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a666bdeb-b0ad-450e-b44a-a05ab6835ace-maas-api-xks-ca\") pod \"a666bdeb-b0ad-450e-b44a-a05ab6835ace\" (UID: \"a666bdeb-b0ad-450e-b44a-a05ab6835ace\") " Oct 31 19:33:43.353086 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.353063 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a666bdeb-b0ad-450e-b44a-a05ab6835ace-kube-api-access-pr2jf\" (UniqueName: \"kubernetes.io/projected/a666bdeb-b0ad-450e-b44a-a05ab6835ace-kube-api-access-pr2jf\") pod \"a666bdeb-b0ad-450e-b44a-a05ab6835ace\" (UID: \"a666bdeb-b0ad-450e-b44a-a05ab6835ace\") " Oct 31 19:33:43.377944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.366805 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a666bdeb-b0ad-450e-b44a-a05ab6835ace-maas-api-openshift-ca" pod "a666bdeb-b0ad-450e-b44a-a05ab6835ace" (UID: "a666bdeb-b0ad-450e-b44a-a05ab6835ace"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:33:43.377944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.367462 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:43.381575 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.381535 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:43.397970 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.387684 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:43.397970 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.394827 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:43.419177 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.419072 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:43.490957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.466998 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a666bdeb-b0ad-450e-b44a-a05ab6835ace-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:33:43.585415 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.584199 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a666bdeb-b0ad-450e-b44a-a05ab6835ace-maas-api-xks-ca" pod "a666bdeb-b0ad-450e-b44a-a05ab6835ace" (UID: "a666bdeb-b0ad-450e-b44a-a05ab6835ace"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:33:43.613467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.613406 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:43.614155 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.614127 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 31 19:33:43.674172 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.674069 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" podUID="be3bf6bf-4863-4489-b924-3de83b589c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 31 19:33:43.674413 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.634493 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:43.674859 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.629096 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:43.675262 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.635858 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a666bdeb-b0ad-450e-b44a-a05ab6835ace-kube-api-access-pr2jf" pod "a666bdeb-b0ad-450e-b44a-a05ab6835ace" (UID: "a666bdeb-b0ad-450e-b44a-a05ab6835ace"). InnerVolumeSpecName "kube-api-access-pr2jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:33:43.705167 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.704454 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a666bdeb-b0ad-450e-b44a-a05ab6835ace-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:33:43.799750 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.799707 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:43.809812 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:33:43.809770 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:33:43.836362 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.836326 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 19:33:43.836938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.836720 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" podUID="71387fb8-12d0-4620-a56c-7b4cd6ba554e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 19:33:43.839556 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:43.839520 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr2jf\" (UniqueName: \"kubernetes.io/projected/a666bdeb-b0ad-450e-b44a-a05ab6835ace-kube-api-access-pr2jf\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:33:44.189126 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:44.186448 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" event={"ID":"1540ff89-ac68-464d-b39f-c01ef4ebd62c","Type":"ContainerStarted","Data":"38382a8c33bdc1e0fdbdd75510fc7669f62aadc2001034561d465318f3aba8ec"} Oct 31 19:33:44.203792 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:44.203758 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" event={"ID":"0ba0cb41-a7be-462c-9e92-7dccedd1d92f","Type":"ContainerStarted","Data":"e707ec070da09d75ca4ecc29b61cb4444cd40b7858c4dc21e9cc5968d43662be"} Oct 31 19:33:44.216270 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:44.216236 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c55970-lql9j" Oct 31 19:33:44.227128 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:44.227089 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c55970-lql9j" event={"ID":"a666bdeb-b0ad-450e-b44a-a05ab6835ace","Type":"ContainerDied","Data":"32d599252c9e7b03acbcca8418691d99041a1879ee5a3d03139a54e094b8a125"} Oct 31 19:33:44.227389 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:44.227367 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32d599252c9e7b03acbcca8418691d99041a1879ee5a3d03139a54e094b8a125" Oct 31 19:33:44.228417 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:44.228397 4910 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 19:33:44.249009 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:44.248968 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:44.281417 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:44.281368 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:44.287785 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:44.287693 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" podStartSLOduration=11.287670286 podStartE2EDuration="11.287670286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:39.397223752 +0000 UTC m=+853.760044124" watchObservedRunningTime="2026-10-31 19:33:44.287670286 +0000 UTC m=+858.650490659" Oct 31 19:33:44.304351 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:44.301402 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:44.385946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:44.330029 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:44.431823 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:44.431775 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:33:44.444042 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:44.439369 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:33:44.472679 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:44.472637 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:33:44.473235 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:44.473192 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:44.523071 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:44.523029 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:33:44.526954 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:44.526926 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:33:44.618719 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:44.610542 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 31 19:33:44.618719 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:44.610605 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" podUID="be3bf6bf-4863-4489-b924-3de83b589c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 31 19:33:44.618719 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:44.617482 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:44.677004 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:44.676955 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:44.901935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:44.901495 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:44.932937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:44.932774 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:45.013104 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:44.963353 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 19:33:45.067594 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.067548 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" podUID="0ba0cb41-a7be-462c-9e92-7dccedd1d92f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 19:33:45.101937 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:33:45.070617 4910 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda666bdeb_b0ad_450e_b44a_a05ab6835ace.slice\": RecentStats: unable to find data in memory cache]" Oct 31 19:33:45.102319 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.102269 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:45.154993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.136235 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:45.257937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.257765 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": context deadline exceeded" Oct 31 19:33:45.259176 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.259136 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:45.259460 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.259436 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:45.259693 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.259656 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:45.284946 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:33:45.264213 4910 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda666bdeb_b0ad_450e_b44a_a05ab6835ace.slice\": RecentStats: unable to find data in memory cache]" Oct 31 19:33:45.285279 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.285236 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:45.299942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.286955 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 19:33:45.299942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.288253 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" podUID="1540ff89-ac68-464d-b39f-c01ef4ebd62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 19:33:45.327520 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.327484 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 19:33:45.328118 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.328088 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" podUID="71387fb8-12d0-4620-a56c-7b4cd6ba554e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 19:33:45.367384 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.367319 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:45.432383 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.432267 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:45.468016 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.444817 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:33:45.489153 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.489102 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:45.492852 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.492813 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:33:45.531651 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.531578 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:45.533084 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.533051 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:45.580944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.579171 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 31 19:33:45.585051 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.584997 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" podUID="be3bf6bf-4863-4489-b924-3de83b589c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 31 19:33:45.619102 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.619056 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:45.842337 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.842299 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 19:33:45.842702 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.842648 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" podUID="71387fb8-12d0-4620-a56c-7b4cd6ba554e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 19:33:45.896617 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.895732 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 19:33:45.896878 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.896846 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" podUID="0ba0cb41-a7be-462c-9e92-7dccedd1d92f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 19:33:45.946742 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:45.946697 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:46.184937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:46.184819 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:46.185316 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:46.185289 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:33:46.526812 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:46.526702 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:46.631992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:46.621680 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": context deadline exceeded" Oct 31 19:33:46.631992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:46.621851 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:33:46.631992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:46.621905 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:33:46.734938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:46.724697 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 19:33:46.734938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:46.724759 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" podUID="1540ff89-ac68-464d-b39f-c01ef4ebd62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 19:33:46.843513 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:46.843450 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:33:46.874435 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:46.874394 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/model-serving-api-75bd89fdfc-lz987" podUID="35fff0bc-d871-4a3b-a6ab-0b6616c80dbd" containerName="server" probeResult="failure" output="Get \"https://10.132.0.44:8443/readyz\": context deadline exceeded" Oct 31 19:33:47.014041 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.013996 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:33:47.215515 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.028286 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:33:47.215970 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.028330 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:33:47.237931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.028359 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:33:47.435935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.072351 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 19:33:47.568001 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.551979 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" podUID="71387fb8-12d0-4620-a56c-7b4cd6ba554e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 19:33:47.568001 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.186014 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 19:33:47.568001 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.552101 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" podUID="0ba0cb41-a7be-462c-9e92-7dccedd1d92f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 19:33:47.568001 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.407652 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 19:33:47.568001 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.552175 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" podUID="1540ff89-ac68-464d-b39f-c01ef4ebd62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 19:33:47.654402 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.654364 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 31 19:33:47.654721 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.654670 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" podUID="be3bf6bf-4863-4489-b924-3de83b589c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 31 19:33:47.693165 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.693130 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 31 19:33:47.727799 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.727755 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" podUID="be3bf6bf-4863-4489-b924-3de83b589c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 31 19:33:47.728080 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:33:47.727255 4910 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.967s" Oct 31 19:33:47.728352 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.728333 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c54dc7ab-r47q6"] Oct 31 19:33:47.729477 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.729453 4910 cpu_manager.go:414] "containerMap: removing container" podUID="a666bdeb-b0ad-450e-b44a-a05ab6835ace" containerName="revoke-keys" Oct 31 19:33:47.729594 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.729582 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a666bdeb-b0ad-450e-b44a-a05ab6835ace" containerName="revoke-keys" Oct 31 19:33:47.729845 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.729832 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="a666bdeb-b0ad-450e-b44a-a05ab6835ace" containerName="revoke-keys" Oct 31 19:33:47.973778 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.972492 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 19:33:47.973778 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.972551 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" podUID="1540ff89-ac68-464d-b39f-c01ef4ebd62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 19:33:47.973778 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.816023 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:47.973778 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.815982 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:47.973778 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.816059 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 19:33:47.973778 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.816737 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded" Oct 31 19:33:47.973778 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.816784 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:33:47.973778 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.816764 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:33:47.973778 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.886700 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded" Oct 31 19:33:47.973778 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.886751 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:47.974495 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.974078 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 19:33:48.029210 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:47.803191 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 19:33:48.064096 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.064058 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 19:33:48.064387 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.064354 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" podUID="0ba0cb41-a7be-462c-9e92-7dccedd1d92f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 19:33:48.109116 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.109065 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" podUID="71387fb8-12d0-4620-a56c-7b4cd6ba554e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 19:33:48.168086 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.168046 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/623cd320-b619-489a-bff1-c63ec2795ecd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c54dc7ab-r47q6\" (UID: \"623cd320-b619-489a-bff1-c63ec2795ecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54dc7ab-r47q6" Oct 31 19:33:48.212971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.212903 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54dc7ab-r47q6" Oct 31 19:33:48.226987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.216972 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdql7\" (UniqueName: \"kubernetes.io/projected/623cd320-b619-489a-bff1-c63ec2795ecd-kube-api-access-zdql7\") pod \"maas-api-revoke-sub-c54dc7ab-r47q6\" (UID: \"623cd320-b619-489a-bff1-c63ec2795ecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54dc7ab-r47q6" Oct 31 19:33:48.242785 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.242576 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:48.244362 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.243170 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/623cd320-b619-489a-bff1-c63ec2795ecd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c54dc7ab-r47q6\" (UID: \"623cd320-b619-489a-bff1-c63ec2795ecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54dc7ab-r47q6" Oct 31 19:33:48.266972 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.257059 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:48.293813 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.293781 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:33:48.294129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.294109 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:33:48.294274 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.294261 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:33:48.294389 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.294378 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:33:48.294516 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.294504 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:33:48.294634 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.294622 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:33:48.294737 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.294726 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:33:48.299963 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.299943 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:33:48.300174 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.300158 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:33:48.374112 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.374077 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c54dc7ab-r47q6"] Oct 31 19:33:48.422937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.399025 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:48.436557 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.436528 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:33:48.464936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.404779 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/623cd320-b619-489a-bff1-c63ec2795ecd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c54dc7ab-r47q6\" (UID: \"623cd320-b619-489a-bff1-c63ec2795ecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54dc7ab-r47q6" Oct 31 19:33:48.470241 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.403829 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/623cd320-b619-489a-bff1-c63ec2795ecd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c54dc7ab-r47q6\" (UID: \"623cd320-b619-489a-bff1-c63ec2795ecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54dc7ab-r47q6" Oct 31 19:33:48.470983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.470943 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdql7\" (UniqueName: \"kubernetes.io/projected/623cd320-b619-489a-bff1-c63ec2795ecd-kube-api-access-zdql7\") pod \"maas-api-revoke-sub-c54dc7ab-r47q6\" (UID: \"623cd320-b619-489a-bff1-c63ec2795ecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54dc7ab-r47q6" Oct 31 19:33:48.471319 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.471296 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/623cd320-b619-489a-bff1-c63ec2795ecd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c54dc7ab-r47q6\" (UID: \"623cd320-b619-489a-bff1-c63ec2795ecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54dc7ab-r47q6" Oct 31 19:33:48.604102 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.604046 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:48.639950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.636366 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:48.668041 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.668011 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 31 19:33:48.668357 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.668319 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" podUID="be3bf6bf-4863-4489-b924-3de83b589c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 31 19:33:48.730065 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.730019 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:48.798382 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.798347 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:48.810281 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.810234 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdql7\" (UniqueName: \"kubernetes.io/projected/623cd320-b619-489a-bff1-c63ec2795ecd-kube-api-access-zdql7\") pod \"maas-api-revoke-sub-c54dc7ab-r47q6\" (UID: \"623cd320-b619-489a-bff1-c63ec2795ecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54dc7ab-r47q6" Oct 31 19:33:48.874945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.874834 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 19:33:48.876257 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.875886 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" podUID="71387fb8-12d0-4620-a56c-7b4cd6ba554e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 19:33:48.895294 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.895246 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 19:33:48.913077 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.913011 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" podUID="0ba0cb41-a7be-462c-9e92-7dccedd1d92f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 19:33:48.913419 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.913383 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:48.926755 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.926725 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 19:33:48.934080 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.934023 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" podUID="1540ff89-ac68-464d-b39f-c01ef4ebd62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 19:33:48.988269 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.986046 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:33:48.988269 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.986447 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 19:33:48.988269 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.986663 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:33:48.988269 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.986885 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:33:48.988269 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.987169 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:33:49.003128 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:48.989276 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:49.180037 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:49.179924 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/623cd320-b619-489a-bff1-c63ec2795ecd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c54dc7ab-r47q6\" (UID: \"623cd320-b619-489a-bff1-c63ec2795ecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54dc7ab-r47q6" Oct 31 19:33:49.272936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:49.245529 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:49.297940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:49.287440 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:49.298542 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:49.298515 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54dc7ab-r47q6" Oct 31 19:33:49.396933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:49.396707 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:33:49.426384 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:49.425456 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:49.560954 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:49.560872 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" podStartSLOduration=16.560836304 podStartE2EDuration="16.560836304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:41.511341717 +0000 UTC m=+855.874162089" watchObservedRunningTime="2026-10-31 19:33:49.560836304 +0000 UTC m=+863.923656679" Oct 31 19:33:49.567803 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:49.567764 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 31 19:33:49.568044 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:49.568011 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" podUID="be3bf6bf-4863-4489-b924-3de83b589c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 31 19:33:49.606086 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:49.606048 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:33:49.607207 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:49.607170 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:33:49.771993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:49.761406 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 19:33:49.771993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:49.761477 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" podUID="0ba0cb41-a7be-462c-9e92-7dccedd1d92f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 19:33:49.773191 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:49.772836 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:49.804937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:49.804618 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" podStartSLOduration=16.80459697 podStartE2EDuration="16.80459697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:44.225515693 +0000 UTC m=+858.588336066" watchObservedRunningTime="2026-10-31 19:33:49.80459697 +0000 UTC m=+864.167417343" Oct 31 19:33:49.941066 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:49.936399 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 19:33:49.941066 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:49.936490 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" podUID="71387fb8-12d0-4620-a56c-7b4cd6ba554e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 19:33:50.415936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:50.367415 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:33:50.415936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:50.367746 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 19:33:50.415936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:50.367954 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:33:50.415936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:50.368154 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:50.415936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:50.368353 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 19:33:50.455622 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:50.429234 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:33:50.455622 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:50.429559 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:50.455622 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:50.430643 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:33:50.495833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:50.484934 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 19:33:50.495833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:50.485003 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" podUID="1540ff89-ac68-464d-b39f-c01ef4ebd62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 19:33:50.560938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:50.555623 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Oct 31 19:33:50.560938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:50.555698 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" podUID="be3bf6bf-4863-4489-b924-3de83b589c8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Oct 31 19:33:50.579170 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:50.579110 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:33:50.666647 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:50.666074 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:50.670458 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:50.670419 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:33:50.845535 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:50.826949 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:33:50.980988 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:50.964327 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 19:33:51.015713 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.015673 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" podUID="71387fb8-12d0-4620-a56c-7b4cd6ba554e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 19:33:51.029952 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.002883 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 19:33:51.029952 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.029525 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" podUID="0ba0cb41-a7be-462c-9e92-7dccedd1d92f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 19:33:51.037941 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:33:51.030629 4910 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.269s" Oct 31 19:33:51.037941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.030829 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:33:51.037941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.030888 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:33:51.037941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.030961 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:33:51.037941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.031015 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:33:51.037941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.031040 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:51.077855 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.077816 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 19:33:51.103956 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.103894 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" podUID="1540ff89-ac68-464d-b39f-c01ef4ebd62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 19:33:51.164314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.164281 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:33:51.166451 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.166410 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:33:51.168219 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.168191 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54dc7ab-r47q6" event={"ID":"623cd320-b619-489a-bff1-c63ec2795ecd","Type":"ContainerStarted","Data":"4a5259956d66c245768939f1213bafaab0f7297f999eaa1f1a566744e1867cf9"} Oct 31 19:33:51.170435 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.170388 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" podStartSLOduration=18.170365019 podStartE2EDuration="18.170365019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:44.522427551 +0000 UTC m=+858.885247938" watchObservedRunningTime="2026-10-31 19:33:51.170365019 +0000 UTC m=+865.533185394" Oct 31 19:33:51.255532 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.255443 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:33:51.374054 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.372546 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:51.374054 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.372960 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:33:51.374054 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.373260 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:33:51.468951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.463545 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:51.487982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.487799 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:33:51.503966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.503492 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:33:51.701503 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.701421 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:33:51.704384 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.704355 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:33:51.708124 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.707975 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:33:51.829940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.828832 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:51.829940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.829145 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:33:51.875061 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.859856 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 19:33:51.875061 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.859923 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" podUID="0ba0cb41-a7be-462c-9e92-7dccedd1d92f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 19:33:51.875061 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.871030 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 19:33:51.875061 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.871079 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" podUID="71387fb8-12d0-4620-a56c-7b4cd6ba554e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 19:33:51.878035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.877926 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 19:33:51.878035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.877974 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" podUID="1540ff89-ac68-464d-b39f-c01ef4ebd62c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 19:33:51.896433 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.896189 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:33:51.896433 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.896235 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-753b8133-xnnxd"] Oct 31 19:33:51.951759 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.951055 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-753b8133-xnnxd"] Oct 31 19:33:51.951759 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.951142 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:33:51.951759 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.951186 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:33:51.951759 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.951224 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:33:51.951759 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.951256 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:33:51.951759 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.951287 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:33:51.951759 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.951319 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:33:51.951759 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.951407 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b8133-xnnxd" Oct 31 19:33:51.980628 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:51.978234 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:33:52.026895 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:52.023950 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv5th\" (UniqueName: \"kubernetes.io/projected/7b5dfe21-5d45-487b-9788-ce7e303d18d9-kube-api-access-mv5th\") pod \"maas-api-revoke-sub-753b8133-xnnxd\" (UID: \"7b5dfe21-5d45-487b-9788-ce7e303d18d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b8133-xnnxd" Oct 31 19:33:52.026895 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:52.024099 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b5dfe21-5d45-487b-9788-ce7e303d18d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-753b8133-xnnxd\" (UID: \"7b5dfe21-5d45-487b-9788-ce7e303d18d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b8133-xnnxd" Oct 31 19:33:52.026895 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:52.025635 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b5dfe21-5d45-487b-9788-ce7e303d18d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-753b8133-xnnxd\" (UID: \"7b5dfe21-5d45-487b-9788-ce7e303d18d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b8133-xnnxd" Oct 31 19:33:52.134938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:52.134298 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b5dfe21-5d45-487b-9788-ce7e303d18d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-753b8133-xnnxd\" (UID: \"7b5dfe21-5d45-487b-9788-ce7e303d18d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b8133-xnnxd" Oct 31 19:33:52.134938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:52.134345 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b5dfe21-5d45-487b-9788-ce7e303d18d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-753b8133-xnnxd\" (UID: \"7b5dfe21-5d45-487b-9788-ce7e303d18d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b8133-xnnxd" Oct 31 19:33:52.134938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:52.134728 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv5th\" (UniqueName: \"kubernetes.io/projected/7b5dfe21-5d45-487b-9788-ce7e303d18d9-kube-api-access-mv5th\") pod \"maas-api-revoke-sub-753b8133-xnnxd\" (UID: \"7b5dfe21-5d45-487b-9788-ce7e303d18d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b8133-xnnxd" Oct 31 19:33:52.139479 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:52.135689 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:52.140080 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:52.139738 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:33:52.140535 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:52.140477 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b5dfe21-5d45-487b-9788-ce7e303d18d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-753b8133-xnnxd\" (UID: \"7b5dfe21-5d45-487b-9788-ce7e303d18d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b8133-xnnxd" Oct 31 19:33:52.199419 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:52.185131 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b5dfe21-5d45-487b-9788-ce7e303d18d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-753b8133-xnnxd\" (UID: \"7b5dfe21-5d45-487b-9788-ce7e303d18d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b8133-xnnxd" Oct 31 19:33:52.199419 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:52.191724 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv5th\" (UniqueName: \"kubernetes.io/projected/7b5dfe21-5d45-487b-9788-ce7e303d18d9-kube-api-access-mv5th\") pod \"maas-api-revoke-sub-753b8133-xnnxd\" (UID: \"7b5dfe21-5d45-487b-9788-ce7e303d18d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b8133-xnnxd" Oct 31 19:33:52.245900 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:52.245813 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54dc7ab-r47q6" event={"ID":"623cd320-b619-489a-bff1-c63ec2795ecd","Type":"ContainerStarted","Data":"d5d3b47675d29dd99f31e705fdc12f4dd76a93c8931d7f5b63eab2edbbaa35ef"} Oct 31 19:33:52.307159 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:52.281141 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c54dc7ab-r47q6"] Oct 31 19:33:52.326640 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:52.325293 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b8133-xnnxd" Oct 31 19:33:52.759945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:52.759154 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Oct 31 19:33:52.759945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:52.759230 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" podUID="0ba0cb41-a7be-462c-9e92-7dccedd1d92f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Oct 31 19:33:52.991373 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:52.991291 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54dc7ab-r47q6" podStartSLOduration=12.991272551 podStartE2EDuration="12.991272551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:52.584616854 +0000 UTC m=+866.947437227" watchObservedRunningTime="2026-10-31 19:33:52.991272551 +0000 UTC m=+867.354092926" Oct 31 19:33:53.000808 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:53.000772 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:53.007018 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:53.006877 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:33:53.087563 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:53.087530 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-753b8133-xnnxd"] Oct 31 19:33:53.259051 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:53.258897 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b8133-xnnxd" event={"ID":"7b5dfe21-5d45-487b-9788-ce7e303d18d9","Type":"ContainerStarted","Data":"351f2b279f3a876e7f59bf3dc8baea0ca8855581f1f9969157d62df63d8fa044"} Oct 31 19:33:53.270201 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:53.268265 4910 generic.go:356] "Generic (PLEG): container finished" podID="623cd320-b619-489a-bff1-c63ec2795ecd" containerID="d5d3b47675d29dd99f31e705fdc12f4dd76a93c8931d7f5b63eab2edbbaa35ef" exitCode=0 Oct 31 19:33:53.270201 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:53.269759 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54dc7ab-r47q6" event={"ID":"623cd320-b619-489a-bff1-c63ec2795ecd","Type":"ContainerDied","Data":"d5d3b47675d29dd99f31e705fdc12f4dd76a93c8931d7f5b63eab2edbbaa35ef"} Oct 31 19:33:53.557755 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:53.557219 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:53.559047 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:53.559026 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:33:53.790542 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:53.790468 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:53.791857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:53.791835 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:33:54.279646 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:54.279615 4910 generic.go:356] "Generic (PLEG): container finished" podID="7b5dfe21-5d45-487b-9788-ce7e303d18d9" containerID="c2f4d0d3073e969052740a03602ab51fac3a9610672b19f5d0f8363267893540" exitCode=0 Oct 31 19:33:54.279778 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:54.279661 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b8133-xnnxd" event={"ID":"7b5dfe21-5d45-487b-9788-ce7e303d18d9","Type":"ContainerDied","Data":"c2f4d0d3073e969052740a03602ab51fac3a9610672b19f5d0f8363267893540"} Oct 31 19:33:54.540528 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:54.540501 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54dc7ab-r47q6" Oct 31 19:33:54.582861 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:54.582826 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/623cd320-b619-489a-bff1-c63ec2795ecd-kube-api-access-zdql7\" (UniqueName: \"kubernetes.io/projected/623cd320-b619-489a-bff1-c63ec2795ecd-kube-api-access-zdql7\") pod \"623cd320-b619-489a-bff1-c63ec2795ecd\" (UID: \"623cd320-b619-489a-bff1-c63ec2795ecd\") " Oct 31 19:33:54.583072 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:54.583027 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/623cd320-b619-489a-bff1-c63ec2795ecd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/623cd320-b619-489a-bff1-c63ec2795ecd-maas-api-openshift-ca\") pod \"623cd320-b619-489a-bff1-c63ec2795ecd\" (UID: \"623cd320-b619-489a-bff1-c63ec2795ecd\") " Oct 31 19:33:54.583174 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:54.583117 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/623cd320-b619-489a-bff1-c63ec2795ecd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/623cd320-b619-489a-bff1-c63ec2795ecd-maas-api-xks-ca\") pod \"623cd320-b619-489a-bff1-c63ec2795ecd\" (UID: \"623cd320-b619-489a-bff1-c63ec2795ecd\") " Oct 31 19:33:54.583461 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:54.583421 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/623cd320-b619-489a-bff1-c63ec2795ecd-maas-api-openshift-ca" pod "623cd320-b619-489a-bff1-c63ec2795ecd" (UID: "623cd320-b619-489a-bff1-c63ec2795ecd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:33:54.584344 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:54.584313 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/623cd320-b619-489a-bff1-c63ec2795ecd-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:33:54.585576 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:54.585536 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/623cd320-b619-489a-bff1-c63ec2795ecd-maas-api-xks-ca" pod "623cd320-b619-489a-bff1-c63ec2795ecd" (UID: "623cd320-b619-489a-bff1-c63ec2795ecd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:33:54.586154 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:54.586126 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/623cd320-b619-489a-bff1-c63ec2795ecd-kube-api-access-zdql7" pod "623cd320-b619-489a-bff1-c63ec2795ecd" (UID: "623cd320-b619-489a-bff1-c63ec2795ecd"). InnerVolumeSpecName "kube-api-access-zdql7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:33:54.686416 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:54.686338 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/623cd320-b619-489a-bff1-c63ec2795ecd-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:33:54.686416 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:54.686371 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdql7\" (UniqueName: \"kubernetes.io/projected/623cd320-b619-489a-bff1-c63ec2795ecd-kube-api-access-zdql7\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:33:55.063180 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.063144 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1858e1c-p5zt6"] Oct 31 19:33:55.064007 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.063984 4910 cpu_manager.go:414] "containerMap: removing container" podUID="623cd320-b619-489a-bff1-c63ec2795ecd" containerName="revoke-keys" Oct 31 19:33:55.064007 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.064007 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="623cd320-b619-489a-bff1-c63ec2795ecd" containerName="revoke-keys" Oct 31 19:33:55.064173 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.064162 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="623cd320-b619-489a-bff1-c63ec2795ecd" containerName="revoke-keys" Oct 31 19:33:55.068185 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.068163 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1858e1c-p5zt6" Oct 31 19:33:55.076651 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.076628 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1858e1c-p5zt6"] Oct 31 19:33:55.107456 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.096652 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0fc3b05-0e60-417e-bd39-ce23ea580292-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1858e1c-p5zt6\" (UID: \"a0fc3b05-0e60-417e-bd39-ce23ea580292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1858e1c-p5zt6" Oct 31 19:33:55.107456 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.096778 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0fc3b05-0e60-417e-bd39-ce23ea580292-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1858e1c-p5zt6\" (UID: \"a0fc3b05-0e60-417e-bd39-ce23ea580292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1858e1c-p5zt6" Oct 31 19:33:55.107456 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.096869 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5whc\" (UniqueName: \"kubernetes.io/projected/a0fc3b05-0e60-417e-bd39-ce23ea580292-kube-api-access-v5whc\") pod \"maas-api-revoke-sub-c1858e1c-p5zt6\" (UID: \"a0fc3b05-0e60-417e-bd39-ce23ea580292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1858e1c-p5zt6" Oct 31 19:33:55.199037 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.198991 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0fc3b05-0e60-417e-bd39-ce23ea580292-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1858e1c-p5zt6\" (UID: \"a0fc3b05-0e60-417e-bd39-ce23ea580292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1858e1c-p5zt6" Oct 31 19:33:55.199189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.199098 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5whc\" (UniqueName: \"kubernetes.io/projected/a0fc3b05-0e60-417e-bd39-ce23ea580292-kube-api-access-v5whc\") pod \"maas-api-revoke-sub-c1858e1c-p5zt6\" (UID: \"a0fc3b05-0e60-417e-bd39-ce23ea580292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1858e1c-p5zt6" Oct 31 19:33:55.199476 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.199459 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0fc3b05-0e60-417e-bd39-ce23ea580292-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1858e1c-p5zt6\" (UID: \"a0fc3b05-0e60-417e-bd39-ce23ea580292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1858e1c-p5zt6" Oct 31 19:33:55.200376 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.200345 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0fc3b05-0e60-417e-bd39-ce23ea580292-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1858e1c-p5zt6\" (UID: \"a0fc3b05-0e60-417e-bd39-ce23ea580292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1858e1c-p5zt6" Oct 31 19:33:55.201742 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.201718 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0fc3b05-0e60-417e-bd39-ce23ea580292-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1858e1c-p5zt6\" (UID: \"a0fc3b05-0e60-417e-bd39-ce23ea580292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1858e1c-p5zt6" Oct 31 19:33:55.214781 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.214738 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5whc\" (UniqueName: \"kubernetes.io/projected/a0fc3b05-0e60-417e-bd39-ce23ea580292-kube-api-access-v5whc\") pod \"maas-api-revoke-sub-c1858e1c-p5zt6\" (UID: \"a0fc3b05-0e60-417e-bd39-ce23ea580292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1858e1c-p5zt6" Oct 31 19:33:55.291602 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.291565 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54dc7ab-r47q6" event={"ID":"623cd320-b619-489a-bff1-c63ec2795ecd","Type":"ContainerDied","Data":"4a5259956d66c245768939f1213bafaab0f7297f999eaa1f1a566744e1867cf9"} Oct 31 19:33:55.291602 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.291602 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54dc7ab-r47q6" Oct 31 19:33:55.291823 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.291653 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a5259956d66c245768939f1213bafaab0f7297f999eaa1f1a566744e1867cf9" Oct 31 19:33:55.387528 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.387490 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1858e1c-p5zt6" Oct 31 19:33:55.513548 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.513302 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b8133-xnnxd" Oct 31 19:33:55.605289 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.605259 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b5dfe21-5d45-487b-9788-ce7e303d18d9-kube-api-access-mv5th\" (UniqueName: \"kubernetes.io/projected/7b5dfe21-5d45-487b-9788-ce7e303d18d9-kube-api-access-mv5th\") pod \"7b5dfe21-5d45-487b-9788-ce7e303d18d9\" (UID: \"7b5dfe21-5d45-487b-9788-ce7e303d18d9\") " Oct 31 19:33:55.605474 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.605349 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b5dfe21-5d45-487b-9788-ce7e303d18d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b5dfe21-5d45-487b-9788-ce7e303d18d9-maas-api-openshift-ca\") pod \"7b5dfe21-5d45-487b-9788-ce7e303d18d9\" (UID: \"7b5dfe21-5d45-487b-9788-ce7e303d18d9\") " Oct 31 19:33:55.605474 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.605402 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b5dfe21-5d45-487b-9788-ce7e303d18d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b5dfe21-5d45-487b-9788-ce7e303d18d9-maas-api-xks-ca\") pod \"7b5dfe21-5d45-487b-9788-ce7e303d18d9\" (UID: \"7b5dfe21-5d45-487b-9788-ce7e303d18d9\") " Oct 31 19:33:55.605747 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.605721 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b5dfe21-5d45-487b-9788-ce7e303d18d9-maas-api-openshift-ca" pod "7b5dfe21-5d45-487b-9788-ce7e303d18d9" (UID: "7b5dfe21-5d45-487b-9788-ce7e303d18d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:33:55.606349 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.606324 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b5dfe21-5d45-487b-9788-ce7e303d18d9-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:33:55.607401 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.607376 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b5dfe21-5d45-487b-9788-ce7e303d18d9-maas-api-xks-ca" pod "7b5dfe21-5d45-487b-9788-ce7e303d18d9" (UID: "7b5dfe21-5d45-487b-9788-ce7e303d18d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:33:55.607570 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.607477 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5dfe21-5d45-487b-9788-ce7e303d18d9-kube-api-access-mv5th" pod "7b5dfe21-5d45-487b-9788-ce7e303d18d9" (UID: "7b5dfe21-5d45-487b-9788-ce7e303d18d9"). InnerVolumeSpecName "kube-api-access-mv5th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:33:55.705749 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.705716 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1858e1c-p5zt6"] Oct 31 19:33:55.706299 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:33:55.706269 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0fc3b05_0e60_417e_bd39_ce23ea580292.slice/crio-d1f352a3a8416ffbe2239252732714b4583609b3bd7faa83ba749660b2c2dc13 WatchSource:0}: Error finding container d1f352a3a8416ffbe2239252732714b4583609b3bd7faa83ba749660b2c2dc13: Status 404 returned error can't find the container with id d1f352a3a8416ffbe2239252732714b4583609b3bd7faa83ba749660b2c2dc13 Oct 31 19:33:55.708454 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.708429 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv5th\" (UniqueName: \"kubernetes.io/projected/7b5dfe21-5d45-487b-9788-ce7e303d18d9-kube-api-access-mv5th\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:33:55.708562 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:55.708458 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b5dfe21-5d45-487b-9788-ce7e303d18d9-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:33:56.304004 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:56.303866 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1858e1c-p5zt6" event={"ID":"a0fc3b05-0e60-417e-bd39-ce23ea580292","Type":"ContainerStarted","Data":"95ea8ea30bfba6477dbd6b2b951be76b32ab6cb8ada82cbab01e5fbe5d8b10ee"} Oct 31 19:33:56.304549 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:56.304525 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1858e1c-p5zt6" event={"ID":"a0fc3b05-0e60-417e-bd39-ce23ea580292","Type":"ContainerStarted","Data":"d1f352a3a8416ffbe2239252732714b4583609b3bd7faa83ba749660b2c2dc13"} Oct 31 19:33:56.305795 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:56.305772 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b8133-xnnxd" Oct 31 19:33:56.305929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:56.305776 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-753b8133-xnnxd" event={"ID":"7b5dfe21-5d45-487b-9788-ce7e303d18d9","Type":"ContainerDied","Data":"351f2b279f3a876e7f59bf3dc8baea0ca8855581f1f9969157d62df63d8fa044"} Oct 31 19:33:56.305929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:56.305895 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="351f2b279f3a876e7f59bf3dc8baea0ca8855581f1f9969157d62df63d8fa044" Oct 31 19:33:56.327563 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:56.327513 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1858e1c-p5zt6" podStartSLOduration=1.327500314 podStartE2EDuration="1.327500314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:56.323846544 +0000 UTC m=+870.686666918" watchObservedRunningTime="2026-10-31 19:33:56.327500314 +0000 UTC m=+870.690320679" Oct 31 19:33:57.316927 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:57.316881 4910 generic.go:356] "Generic (PLEG): container finished" podID="a0fc3b05-0e60-417e-bd39-ce23ea580292" containerID="95ea8ea30bfba6477dbd6b2b951be76b32ab6cb8ada82cbab01e5fbe5d8b10ee" exitCode=0 Oct 31 19:33:57.317291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:57.316963 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1858e1c-p5zt6" event={"ID":"a0fc3b05-0e60-417e-bd39-ce23ea580292","Type":"ContainerDied","Data":"95ea8ea30bfba6477dbd6b2b951be76b32ab6cb8ada82cbab01e5fbe5d8b10ee"} Oct 31 19:33:58.507409 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.507381 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1858e1c-p5zt6" Oct 31 19:33:58.545423 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.545388 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d470d6-95bd6"] Oct 31 19:33:58.546512 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.546491 4910 cpu_manager.go:414] "containerMap: removing container" podUID="a0fc3b05-0e60-417e-bd39-ce23ea580292" containerName="revoke-keys" Oct 31 19:33:58.546512 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.546516 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0fc3b05-0e60-417e-bd39-ce23ea580292" containerName="revoke-keys" Oct 31 19:33:58.546640 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.546560 4910 cpu_manager.go:414] "containerMap: removing container" podUID="7b5dfe21-5d45-487b-9788-ce7e303d18d9" containerName="revoke-keys" Oct 31 19:33:58.546640 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.546568 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b5dfe21-5d45-487b-9788-ce7e303d18d9" containerName="revoke-keys" Oct 31 19:33:58.546761 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.546748 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0fc3b05-0e60-417e-bd39-ce23ea580292" containerName="revoke-keys" Oct 31 19:33:58.546792 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.546771 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b5dfe21-5d45-487b-9788-ce7e303d18d9" containerName="revoke-keys" Oct 31 19:33:58.551279 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.551256 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d470d6-95bd6" Oct 31 19:33:58.561170 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.561139 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0fc3b05-0e60-417e-bd39-ce23ea580292-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0fc3b05-0e60-417e-bd39-ce23ea580292-maas-api-xks-ca\") pod \"a0fc3b05-0e60-417e-bd39-ce23ea580292\" (UID: \"a0fc3b05-0e60-417e-bd39-ce23ea580292\") " Oct 31 19:33:58.561312 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.561209 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0fc3b05-0e60-417e-bd39-ce23ea580292-kube-api-access-v5whc\" (UniqueName: \"kubernetes.io/projected/a0fc3b05-0e60-417e-bd39-ce23ea580292-kube-api-access-v5whc\") pod \"a0fc3b05-0e60-417e-bd39-ce23ea580292\" (UID: \"a0fc3b05-0e60-417e-bd39-ce23ea580292\") " Oct 31 19:33:58.561312 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.561280 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0fc3b05-0e60-417e-bd39-ce23ea580292-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0fc3b05-0e60-417e-bd39-ce23ea580292-maas-api-openshift-ca\") pod \"a0fc3b05-0e60-417e-bd39-ce23ea580292\" (UID: \"a0fc3b05-0e60-417e-bd39-ce23ea580292\") " Oct 31 19:33:58.562578 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.562547 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0fc3b05-0e60-417e-bd39-ce23ea580292-maas-api-openshift-ca" pod "a0fc3b05-0e60-417e-bd39-ce23ea580292" (UID: "a0fc3b05-0e60-417e-bd39-ce23ea580292"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:33:58.566562 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.566509 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0fc3b05-0e60-417e-bd39-ce23ea580292-maas-api-xks-ca" pod "a0fc3b05-0e60-417e-bd39-ce23ea580292" (UID: "a0fc3b05-0e60-417e-bd39-ce23ea580292"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:33:58.568686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.568625 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0fc3b05-0e60-417e-bd39-ce23ea580292-kube-api-access-v5whc" pod "a0fc3b05-0e60-417e-bd39-ce23ea580292" (UID: "a0fc3b05-0e60-417e-bd39-ce23ea580292"). InnerVolumeSpecName "kube-api-access-v5whc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:33:58.572285 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.572258 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d470d6-95bd6"] Oct 31 19:33:58.664040 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.664000 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dde4aa8-330b-4bdd-93c3-58ae26d99dfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08d470d6-95bd6\" (UID: \"4dde4aa8-330b-4bdd-93c3-58ae26d99dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d470d6-95bd6" Oct 31 19:33:58.664040 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.664046 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dde4aa8-330b-4bdd-93c3-58ae26d99dfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08d470d6-95bd6\" (UID: \"4dde4aa8-330b-4bdd-93c3-58ae26d99dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d470d6-95bd6" Oct 31 19:33:58.664267 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.664237 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9cc8\" (UniqueName: \"kubernetes.io/projected/4dde4aa8-330b-4bdd-93c3-58ae26d99dfe-kube-api-access-l9cc8\") pod \"maas-api-revoke-sub-08d470d6-95bd6\" (UID: \"4dde4aa8-330b-4bdd-93c3-58ae26d99dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d470d6-95bd6" Oct 31 19:33:58.664546 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.664529 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0fc3b05-0e60-417e-bd39-ce23ea580292-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:33:58.664598 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.664553 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5whc\" (UniqueName: \"kubernetes.io/projected/a0fc3b05-0e60-417e-bd39-ce23ea580292-kube-api-access-v5whc\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:33:58.664598 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.664571 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0fc3b05-0e60-417e-bd39-ce23ea580292-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:33:58.767301 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.767267 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dde4aa8-330b-4bdd-93c3-58ae26d99dfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08d470d6-95bd6\" (UID: \"4dde4aa8-330b-4bdd-93c3-58ae26d99dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d470d6-95bd6" Oct 31 19:33:58.767629 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.767607 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dde4aa8-330b-4bdd-93c3-58ae26d99dfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08d470d6-95bd6\" (UID: \"4dde4aa8-330b-4bdd-93c3-58ae26d99dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d470d6-95bd6" Oct 31 19:33:58.767858 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.767843 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9cc8\" (UniqueName: \"kubernetes.io/projected/4dde4aa8-330b-4bdd-93c3-58ae26d99dfe-kube-api-access-l9cc8\") pod \"maas-api-revoke-sub-08d470d6-95bd6\" (UID: \"4dde4aa8-330b-4bdd-93c3-58ae26d99dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d470d6-95bd6" Oct 31 19:33:58.768441 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.768405 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dde4aa8-330b-4bdd-93c3-58ae26d99dfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08d470d6-95bd6\" (UID: \"4dde4aa8-330b-4bdd-93c3-58ae26d99dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d470d6-95bd6" Oct 31 19:33:58.775207 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.772294 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dde4aa8-330b-4bdd-93c3-58ae26d99dfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08d470d6-95bd6\" (UID: \"4dde4aa8-330b-4bdd-93c3-58ae26d99dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d470d6-95bd6" Oct 31 19:33:58.801010 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.794542 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9cc8\" (UniqueName: \"kubernetes.io/projected/4dde4aa8-330b-4bdd-93c3-58ae26d99dfe-kube-api-access-l9cc8\") pod \"maas-api-revoke-sub-08d470d6-95bd6\" (UID: \"4dde4aa8-330b-4bdd-93c3-58ae26d99dfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d470d6-95bd6" Oct 31 19:33:58.921320 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:58.921226 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d470d6-95bd6" Oct 31 19:33:59.209698 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:33:59.209661 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dde4aa8_330b_4bdd_93c3_58ae26d99dfe.slice/crio-c89e53efe66c571f0851f7db46a8948fd98c6c259652167a07ab162ac251b127 WatchSource:0}: Error finding container c89e53efe66c571f0851f7db46a8948fd98c6c259652167a07ab162ac251b127: Status 404 returned error can't find the container with id c89e53efe66c571f0851f7db46a8948fd98c6c259652167a07ab162ac251b127 Oct 31 19:33:59.210861 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:59.210837 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d470d6-95bd6"] Oct 31 19:33:59.336578 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:59.336547 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1858e1c-p5zt6" Oct 31 19:33:59.336578 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:59.336571 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1858e1c-p5zt6" event={"ID":"a0fc3b05-0e60-417e-bd39-ce23ea580292","Type":"ContainerDied","Data":"d1f352a3a8416ffbe2239252732714b4583609b3bd7faa83ba749660b2c2dc13"} Oct 31 19:33:59.336819 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:59.336616 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1f352a3a8416ffbe2239252732714b4583609b3bd7faa83ba749660b2c2dc13" Oct 31 19:33:59.338120 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:33:59.338094 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d470d6-95bd6" event={"ID":"4dde4aa8-330b-4bdd-93c3-58ae26d99dfe","Type":"ContainerStarted","Data":"c89e53efe66c571f0851f7db46a8948fd98c6c259652167a07ab162ac251b127"} Oct 31 19:34:00.363583 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:00.363545 4910 generic.go:356] "Generic (PLEG): container finished" podID="4dde4aa8-330b-4bdd-93c3-58ae26d99dfe" containerID="e3ead7565de785b5853be4d8d7152ffec01086847aa022a6ec11f236468a0db9" exitCode=0 Oct 31 19:34:00.364341 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:00.363814 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d470d6-95bd6" event={"ID":"4dde4aa8-330b-4bdd-93c3-58ae26d99dfe","Type":"ContainerDied","Data":"e3ead7565de785b5853be4d8d7152ffec01086847aa022a6ec11f236468a0db9"} Oct 31 19:34:01.358630 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:01.358592 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dd2ce39-pr7sr"] Oct 31 19:34:01.381943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:01.380109 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dd2ce39-pr7sr"] Oct 31 19:34:01.381943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:01.380271 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd2ce39-pr7sr" Oct 31 19:34:01.513364 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:01.513324 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3094a11-6c88-4340-b63b-03d19ad814dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dd2ce39-pr7sr\" (UID: \"f3094a11-6c88-4340-b63b-03d19ad814dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd2ce39-pr7sr" Oct 31 19:34:01.513551 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:01.513464 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bmg5\" (UniqueName: \"kubernetes.io/projected/f3094a11-6c88-4340-b63b-03d19ad814dd-kube-api-access-4bmg5\") pod \"maas-api-revoke-sub-7dd2ce39-pr7sr\" (UID: \"f3094a11-6c88-4340-b63b-03d19ad814dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd2ce39-pr7sr" Oct 31 19:34:01.513796 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:01.513776 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3094a11-6c88-4340-b63b-03d19ad814dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dd2ce39-pr7sr\" (UID: \"f3094a11-6c88-4340-b63b-03d19ad814dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd2ce39-pr7sr" Oct 31 19:34:01.617019 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:01.615823 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3094a11-6c88-4340-b63b-03d19ad814dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dd2ce39-pr7sr\" (UID: \"f3094a11-6c88-4340-b63b-03d19ad814dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd2ce39-pr7sr" Oct 31 19:34:01.617019 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:01.616014 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3094a11-6c88-4340-b63b-03d19ad814dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dd2ce39-pr7sr\" (UID: \"f3094a11-6c88-4340-b63b-03d19ad814dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd2ce39-pr7sr" Oct 31 19:34:01.617019 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:01.616081 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bmg5\" (UniqueName: \"kubernetes.io/projected/f3094a11-6c88-4340-b63b-03d19ad814dd-kube-api-access-4bmg5\") pod \"maas-api-revoke-sub-7dd2ce39-pr7sr\" (UID: \"f3094a11-6c88-4340-b63b-03d19ad814dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd2ce39-pr7sr" Oct 31 19:34:01.617288 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:01.617174 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3094a11-6c88-4340-b63b-03d19ad814dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dd2ce39-pr7sr\" (UID: \"f3094a11-6c88-4340-b63b-03d19ad814dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd2ce39-pr7sr" Oct 31 19:34:01.619905 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:01.619873 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3094a11-6c88-4340-b63b-03d19ad814dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dd2ce39-pr7sr\" (UID: \"f3094a11-6c88-4340-b63b-03d19ad814dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd2ce39-pr7sr" Oct 31 19:34:01.645695 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:01.645648 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bmg5\" (UniqueName: \"kubernetes.io/projected/f3094a11-6c88-4340-b63b-03d19ad814dd-kube-api-access-4bmg5\") pod \"maas-api-revoke-sub-7dd2ce39-pr7sr\" (UID: \"f3094a11-6c88-4340-b63b-03d19ad814dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd2ce39-pr7sr" Oct 31 19:34:01.655333 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:01.655299 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d470d6-95bd6" Oct 31 19:34:01.704605 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:01.704567 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd2ce39-pr7sr" Oct 31 19:34:01.726663 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:01.726579 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dde4aa8-330b-4bdd-93c3-58ae26d99dfe-kube-api-access-l9cc8\" (UniqueName: \"kubernetes.io/projected/4dde4aa8-330b-4bdd-93c3-58ae26d99dfe-kube-api-access-l9cc8\") pod \"4dde4aa8-330b-4bdd-93c3-58ae26d99dfe\" (UID: \"4dde4aa8-330b-4bdd-93c3-58ae26d99dfe\") " Oct 31 19:34:01.727085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:01.726739 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dde4aa8-330b-4bdd-93c3-58ae26d99dfe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dde4aa8-330b-4bdd-93c3-58ae26d99dfe-maas-api-openshift-ca\") pod \"4dde4aa8-330b-4bdd-93c3-58ae26d99dfe\" (UID: \"4dde4aa8-330b-4bdd-93c3-58ae26d99dfe\") " Oct 31 19:34:01.727085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:01.726785 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dde4aa8-330b-4bdd-93c3-58ae26d99dfe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dde4aa8-330b-4bdd-93c3-58ae26d99dfe-maas-api-xks-ca\") pod \"4dde4aa8-330b-4bdd-93c3-58ae26d99dfe\" (UID: \"4dde4aa8-330b-4bdd-93c3-58ae26d99dfe\") " Oct 31 19:34:01.730339 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:01.730307 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dde4aa8-330b-4bdd-93c3-58ae26d99dfe-maas-api-openshift-ca" pod "4dde4aa8-330b-4bdd-93c3-58ae26d99dfe" (UID: "4dde4aa8-330b-4bdd-93c3-58ae26d99dfe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:34:01.731964 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:01.731904 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dde4aa8-330b-4bdd-93c3-58ae26d99dfe-kube-api-access-l9cc8" pod "4dde4aa8-330b-4bdd-93c3-58ae26d99dfe" (UID: "4dde4aa8-330b-4bdd-93c3-58ae26d99dfe"). InnerVolumeSpecName "kube-api-access-l9cc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:34:01.733845 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:01.733812 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dde4aa8-330b-4bdd-93c3-58ae26d99dfe-maas-api-xks-ca" pod "4dde4aa8-330b-4bdd-93c3-58ae26d99dfe" (UID: "4dde4aa8-330b-4bdd-93c3-58ae26d99dfe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:34:01.829824 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:01.829785 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dde4aa8-330b-4bdd-93c3-58ae26d99dfe-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:01.829824 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:01.829827 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dde4aa8-330b-4bdd-93c3-58ae26d99dfe-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:01.830055 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:01.829842 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9cc8\" (UniqueName: \"kubernetes.io/projected/4dde4aa8-330b-4bdd-93c3-58ae26d99dfe-kube-api-access-l9cc8\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:02.014813 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:34:02.014781 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3094a11_6c88_4340_b63b_03d19ad814dd.slice/crio-3686f1061cac400d40077f68fe255608bd6e062496fad082cfd523e6eea56db9 WatchSource:0}: Error finding container 3686f1061cac400d40077f68fe255608bd6e062496fad082cfd523e6eea56db9: Status 404 returned error can't find the container with id 3686f1061cac400d40077f68fe255608bd6e062496fad082cfd523e6eea56db9 Oct 31 19:34:02.017055 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:02.017029 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dd2ce39-pr7sr"] Oct 31 19:34:02.406525 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:02.406485 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d470d6-95bd6" event={"ID":"4dde4aa8-330b-4bdd-93c3-58ae26d99dfe","Type":"ContainerDied","Data":"c89e53efe66c571f0851f7db46a8948fd98c6c259652167a07ab162ac251b127"} Oct 31 19:34:02.407242 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:02.407220 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c89e53efe66c571f0851f7db46a8948fd98c6c259652167a07ab162ac251b127" Oct 31 19:34:02.407518 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:02.407499 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08d470d6-95bd6" Oct 31 19:34:02.415009 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:02.414888 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd2ce39-pr7sr" event={"ID":"f3094a11-6c88-4340-b63b-03d19ad814dd","Type":"ContainerStarted","Data":"3cb74bdac143ca1e096dd4e0df2a4d2d30fa784688c3d76864cc2e499f40d728"} Oct 31 19:34:02.415009 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:02.414982 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd2ce39-pr7sr" event={"ID":"f3094a11-6c88-4340-b63b-03d19ad814dd","Type":"ContainerStarted","Data":"3686f1061cac400d40077f68fe255608bd6e062496fad082cfd523e6eea56db9"} Oct 31 19:34:02.461810 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:02.461741 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd2ce39-pr7sr" podStartSLOduration=1.461720865 podStartE2EDuration="1.461720865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:02.454952792 +0000 UTC m=+876.817773167" watchObservedRunningTime="2026-10-31 19:34:02.461720865 +0000 UTC m=+876.824541239" Oct 31 19:34:03.431905 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:03.431860 4910 generic.go:356] "Generic (PLEG): container finished" podID="f3094a11-6c88-4340-b63b-03d19ad814dd" containerID="3cb74bdac143ca1e096dd4e0df2a4d2d30fa784688c3d76864cc2e499f40d728" exitCode=0 Oct 31 19:34:03.432438 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:03.431955 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd2ce39-pr7sr" event={"ID":"f3094a11-6c88-4340-b63b-03d19ad814dd","Type":"ContainerDied","Data":"3cb74bdac143ca1e096dd4e0df2a4d2d30fa784688c3d76864cc2e499f40d728"} Oct 31 19:34:04.622690 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:04.622664 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd2ce39-pr7sr" Oct 31 19:34:04.782546 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:04.780715 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3094a11-6c88-4340-b63b-03d19ad814dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3094a11-6c88-4340-b63b-03d19ad814dd-maas-api-openshift-ca\") pod \"f3094a11-6c88-4340-b63b-03d19ad814dd\" (UID: \"f3094a11-6c88-4340-b63b-03d19ad814dd\") " Oct 31 19:34:04.782546 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:04.781038 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3094a11-6c88-4340-b63b-03d19ad814dd-kube-api-access-4bmg5\" (UniqueName: \"kubernetes.io/projected/f3094a11-6c88-4340-b63b-03d19ad814dd-kube-api-access-4bmg5\") pod \"f3094a11-6c88-4340-b63b-03d19ad814dd\" (UID: \"f3094a11-6c88-4340-b63b-03d19ad814dd\") " Oct 31 19:34:04.782546 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:04.781122 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3094a11-6c88-4340-b63b-03d19ad814dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3094a11-6c88-4340-b63b-03d19ad814dd-maas-api-xks-ca\") pod \"f3094a11-6c88-4340-b63b-03d19ad814dd\" (UID: \"f3094a11-6c88-4340-b63b-03d19ad814dd\") " Oct 31 19:34:04.782546 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:04.782453 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3094a11-6c88-4340-b63b-03d19ad814dd-maas-api-openshift-ca" pod "f3094a11-6c88-4340-b63b-03d19ad814dd" (UID: "f3094a11-6c88-4340-b63b-03d19ad814dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:34:04.798957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:04.782721 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3094a11-6c88-4340-b63b-03d19ad814dd-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:04.812502 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:04.804509 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3094a11-6c88-4340-b63b-03d19ad814dd-maas-api-xks-ca" pod "f3094a11-6c88-4340-b63b-03d19ad814dd" (UID: "f3094a11-6c88-4340-b63b-03d19ad814dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:34:04.816789 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:04.816683 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3094a11-6c88-4340-b63b-03d19ad814dd-kube-api-access-4bmg5" pod "f3094a11-6c88-4340-b63b-03d19ad814dd" (UID: "f3094a11-6c88-4340-b63b-03d19ad814dd"). InnerVolumeSpecName "kube-api-access-4bmg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:34:04.884028 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:04.883995 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bmg5\" (UniqueName: \"kubernetes.io/projected/f3094a11-6c88-4340-b63b-03d19ad814dd-kube-api-access-4bmg5\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:04.884028 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:04.884023 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3094a11-6c88-4340-b63b-03d19ad814dd-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:05.456998 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:05.456962 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd2ce39-pr7sr" event={"ID":"f3094a11-6c88-4340-b63b-03d19ad814dd","Type":"ContainerDied","Data":"3686f1061cac400d40077f68fe255608bd6e062496fad082cfd523e6eea56db9"} Oct 31 19:34:05.456998 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:05.457025 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3686f1061cac400d40077f68fe255608bd6e062496fad082cfd523e6eea56db9" Oct 31 19:34:05.457250 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:05.457053 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd2ce39-pr7sr" Oct 31 19:34:07.456299 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:07.456263 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d508d5-jk8c5"] Oct 31 19:34:07.457026 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:07.457008 4910 cpu_manager.go:414] "containerMap: removing container" podUID="f3094a11-6c88-4340-b63b-03d19ad814dd" containerName="revoke-keys" Oct 31 19:34:07.457026 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:07.457025 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3094a11-6c88-4340-b63b-03d19ad814dd" containerName="revoke-keys" Oct 31 19:34:07.457165 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:07.457044 4910 cpu_manager.go:414] "containerMap: removing container" podUID="4dde4aa8-330b-4bdd-93c3-58ae26d99dfe" containerName="revoke-keys" Oct 31 19:34:07.457165 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:07.457049 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dde4aa8-330b-4bdd-93c3-58ae26d99dfe" containerName="revoke-keys" Oct 31 19:34:07.457266 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:07.457193 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3094a11-6c88-4340-b63b-03d19ad814dd" containerName="revoke-keys" Oct 31 19:34:07.457266 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:07.457204 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dde4aa8-330b-4bdd-93c3-58ae26d99dfe" containerName="revoke-keys" Oct 31 19:34:07.462059 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:07.462032 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d508d5-jk8c5" Oct 31 19:34:07.469033 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:07.469007 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:34:07.469234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:07.469218 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:34:07.471405 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:07.471379 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d508d5-jk8c5"] Oct 31 19:34:07.535807 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:07.535764 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgvwd\" (UniqueName: \"kubernetes.io/projected/faeecdd8-67fc-4249-ba29-ab06d605de57-kube-api-access-mgvwd\") pod \"maas-api-revoke-sub-55d508d5-jk8c5\" (UID: \"faeecdd8-67fc-4249-ba29-ab06d605de57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d508d5-jk8c5" Oct 31 19:34:07.536019 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:07.535901 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faeecdd8-67fc-4249-ba29-ab06d605de57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55d508d5-jk8c5\" (UID: \"faeecdd8-67fc-4249-ba29-ab06d605de57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d508d5-jk8c5" Oct 31 19:34:07.536077 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:07.536039 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faeecdd8-67fc-4249-ba29-ab06d605de57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55d508d5-jk8c5\" (UID: \"faeecdd8-67fc-4249-ba29-ab06d605de57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d508d5-jk8c5" Oct 31 19:34:07.637879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:07.637840 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faeecdd8-67fc-4249-ba29-ab06d605de57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55d508d5-jk8c5\" (UID: \"faeecdd8-67fc-4249-ba29-ab06d605de57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d508d5-jk8c5" Oct 31 19:34:07.638094 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:07.638000 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgvwd\" (UniqueName: \"kubernetes.io/projected/faeecdd8-67fc-4249-ba29-ab06d605de57-kube-api-access-mgvwd\") pod \"maas-api-revoke-sub-55d508d5-jk8c5\" (UID: \"faeecdd8-67fc-4249-ba29-ab06d605de57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d508d5-jk8c5" Oct 31 19:34:07.638094 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:07.638086 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faeecdd8-67fc-4249-ba29-ab06d605de57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55d508d5-jk8c5\" (UID: \"faeecdd8-67fc-4249-ba29-ab06d605de57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d508d5-jk8c5" Oct 31 19:34:07.638774 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:07.638751 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faeecdd8-67fc-4249-ba29-ab06d605de57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55d508d5-jk8c5\" (UID: \"faeecdd8-67fc-4249-ba29-ab06d605de57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d508d5-jk8c5" Oct 31 19:34:07.640343 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:07.640321 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faeecdd8-67fc-4249-ba29-ab06d605de57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55d508d5-jk8c5\" (UID: \"faeecdd8-67fc-4249-ba29-ab06d605de57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d508d5-jk8c5" Oct 31 19:34:07.651061 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:07.651034 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgvwd\" (UniqueName: \"kubernetes.io/projected/faeecdd8-67fc-4249-ba29-ab06d605de57-kube-api-access-mgvwd\") pod \"maas-api-revoke-sub-55d508d5-jk8c5\" (UID: \"faeecdd8-67fc-4249-ba29-ab06d605de57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d508d5-jk8c5" Oct 31 19:34:07.782998 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:07.782874 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d508d5-jk8c5" Oct 31 19:34:08.083200 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:08.083173 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d508d5-jk8c5"] Oct 31 19:34:08.490689 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:08.490657 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d508d5-jk8c5" event={"ID":"faeecdd8-67fc-4249-ba29-ab06d605de57","Type":"ContainerStarted","Data":"33e3a4b3860a0f884ff227659651a566fe1542190911e67cdf70bbfab0273e7e"} Oct 31 19:34:08.491124 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:08.490709 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d508d5-jk8c5" event={"ID":"faeecdd8-67fc-4249-ba29-ab06d605de57","Type":"ContainerStarted","Data":"0f82e46d7d463b5832307c0db523b1f8f04dfd69c0cbc4d916233c41c38e08c1"} Oct 31 19:34:08.515283 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:08.515230 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d508d5-jk8c5" podStartSLOduration=1.515216958 podStartE2EDuration="1.515216958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:08.509502002 +0000 UTC m=+882.872322375" watchObservedRunningTime="2026-10-31 19:34:08.515216958 +0000 UTC m=+882.878037329" Oct 31 19:34:09.506143 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:09.506084 4910 generic.go:356] "Generic (PLEG): container finished" podID="faeecdd8-67fc-4249-ba29-ab06d605de57" containerID="33e3a4b3860a0f884ff227659651a566fe1542190911e67cdf70bbfab0273e7e" exitCode=0 Oct 31 19:34:09.506691 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:09.506278 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d508d5-jk8c5" event={"ID":"faeecdd8-67fc-4249-ba29-ab06d605de57","Type":"ContainerDied","Data":"33e3a4b3860a0f884ff227659651a566fe1542190911e67cdf70bbfab0273e7e"} Oct 31 19:34:10.673015 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:10.672968 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7011-23b52b"] Oct 31 19:34:10.809086 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:10.809030 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:10.854337 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:10.816234 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-23b52b" Oct 31 19:34:10.879935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:10.816279 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:10.915003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:10.908493 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4pmld\"" Oct 31 19:34:10.915003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:10.819839 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7011-23b52b"] Oct 31 19:34:10.915003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:10.880267 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:34:10.940937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:10.816315 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:34:10.969023 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:10.968983 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p56tl\" (UniqueName: \"kubernetes.io/projected/a2913723-bd1a-4797-af4f-c726356e2ed1-kube-api-access-p56tl\") pod \"test-curl-7011-23b52b\" (UID: \"a2913723-bd1a-4797-af4f-c726356e2ed1\") " pod="opendatahub/test-curl-7011-23b52b" Oct 31 19:34:10.987768 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:10.987719 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:34:10.992003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:10.988053 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:11.080329 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:11.079645 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p56tl\" (UniqueName: \"kubernetes.io/projected/a2913723-bd1a-4797-af4f-c726356e2ed1-kube-api-access-p56tl\") pod \"test-curl-7011-23b52b\" (UID: \"a2913723-bd1a-4797-af4f-c726356e2ed1\") " pod="opendatahub/test-curl-7011-23b52b" Oct 31 19:34:11.146937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:11.142823 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p56tl\" (UniqueName: \"kubernetes.io/projected/a2913723-bd1a-4797-af4f-c726356e2ed1-kube-api-access-p56tl\") pod \"test-curl-7011-23b52b\" (UID: \"a2913723-bd1a-4797-af4f-c726356e2ed1\") " pod="opendatahub/test-curl-7011-23b52b" Oct 31 19:34:11.175993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:11.165021 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:11.285939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:11.268038 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:11.308948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:11.305022 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 19:34:11.369088 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:11.369035 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:11.387102 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:11.387041 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:11.389033 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:11.389007 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-23b52b" Oct 31 19:34:11.389567 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:11.389544 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:34:11.624121 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:11.624063 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:34:11.892953 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:11.868043 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:11.922993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:11.910695 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:11.922993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:11.915003 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:11.950228 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:11.950177 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:11.995835 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:11.995786 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:12.004984 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:11.996447 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:34:12.026322 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:12.026061 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:12.059938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:12.059023 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:12.117008 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:12.106373 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:34:12.117008 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:12.106479 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:34:12.117008 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:12.106515 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:34:12.117008 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:12.106546 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:34:12.117008 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:12.106575 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:34:12.117008 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:12.106776 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:34:12.318014 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:12.281468 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:34:12.318014 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:12.281992 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:34:12.318014 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:12.312989 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 19:34:12.318014 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:12.313843 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:34:12.366936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:12.335661 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:12.423121 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:12.386987 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:12.423121 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:12.388167 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:34:12.424157 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:12.424110 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:34:12.431936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:12.424803 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:34:12.459716 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:12.425752 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:34:12.467937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:12.464467 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:34:12.516010 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:12.509512 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:34:12.937955 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:12.910997 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 19:34:12.937955 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:12.912388 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:12.941205 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:12.940328 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:13.069003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:13.002552 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:13.069003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:13.038271 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:34:13.069003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:13.038595 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:34:13.192272 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:13.172678 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:13.368995 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:13.362987 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:13.468038 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:13.438444 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:34:13.500247 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:13.500215 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:34:13.755288 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:13.755158 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:13.770941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:13.765838 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:13.914460 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:13.780216 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:34:13.914460 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:13.848518 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" podUID="f9561493-df06-4f24-878e-f5d1255ac41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:13.914460 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:13.888266 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:14.068872 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:14.040110 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:14.068872 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:14.056113 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:14.249029 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:14.083990 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 19:34:14.343211 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:14.343108 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:14.378455 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:14.351221 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:14.399890 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:14.383476 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:34:14.399890 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:14.383531 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" podUID="ce5603fa-39d7-4705-a0e3-85297ad53a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:14.399890 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:14.383590 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:34:14.453414 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:14.453371 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:14.579599 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:14.579540 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:14.802356 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:14.802303 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:14.828270 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:14.828219 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:34:14.901142 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:34:14.744396 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:34:14.939932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:14.777442 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:34:14.940216 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:14.828805 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:15.243700 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.006019 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:34:15.243700 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.006084 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:15.243700 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.142407 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:15.243700 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.205132 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:15.290040 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.290008 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:34:15.290293 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.290262 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" podUID="0a24bfa8-34c0-4825-876a-c2fec6019829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:15.290460 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.290439 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 19:34:15.347830 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.345579 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:15.347830 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.345931 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:15.350057 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.350025 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:15.373561 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:34:15.368557 4910 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.568s" Oct 31 19:34:15.373561 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.368701 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:34:15.373561 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.368785 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:34:15.373561 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.368846 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:34:15.397264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.397206 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:15.456189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.456137 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:15.458079 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.458040 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:34:15.458288 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.458262 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" podUID="005cc55c-8e13-47d1-aa18-ddb1fb79432c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:15.585779 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.584238 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:15.585779 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.584738 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d508d5-jk8c5" Oct 31 19:34:15.759949 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.759839 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faeecdd8-67fc-4249-ba29-ab06d605de57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faeecdd8-67fc-4249-ba29-ab06d605de57-maas-api-openshift-ca\") pod \"faeecdd8-67fc-4249-ba29-ab06d605de57\" (UID: \"faeecdd8-67fc-4249-ba29-ab06d605de57\") " Oct 31 19:34:15.760468 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.760438 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faeecdd8-67fc-4249-ba29-ab06d605de57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faeecdd8-67fc-4249-ba29-ab06d605de57-maas-api-xks-ca\") pod \"faeecdd8-67fc-4249-ba29-ab06d605de57\" (UID: \"faeecdd8-67fc-4249-ba29-ab06d605de57\") " Oct 31 19:34:16.255198 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:16.255151 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faeecdd8-67fc-4249-ba29-ab06d605de57-kube-api-access-mgvwd\" (UniqueName: \"kubernetes.io/projected/faeecdd8-67fc-4249-ba29-ab06d605de57-kube-api-access-mgvwd\") pod \"faeecdd8-67fc-4249-ba29-ab06d605de57\" (UID: \"faeecdd8-67fc-4249-ba29-ab06d605de57\") " Oct 31 19:34:16.328184 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.972059 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:16.328858 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:16.154025 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:16.354434 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:16.237539 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:34:16.391018 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:16.358256 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:16.394090 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:16.394049 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:16.489652 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:16.281015 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:34:16.493035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:16.477286 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:34:16.503237 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:16.477324 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:34:16.526848 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:16.502437 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:34:16.528205 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.787127 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faeecdd8-67fc-4249-ba29-ab06d605de57-maas-api-openshift-ca" pod "faeecdd8-67fc-4249-ba29-ab06d605de57" (UID: "faeecdd8-67fc-4249-ba29-ab06d605de57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:34:16.528399 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.898019 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:34:16.545056 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:16.545001 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" podUID="6ab4bc46-8c70-4f5a-8e0d-f852190f8c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:16.545322 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.898081 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:34:16.545461 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:16.545433 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" podUID="09020e9b-b4e8-4cbc-946e-73f3f7e8fc94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:16.545573 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:15.925072 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:34:16.545690 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:16.545669 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" podUID="638cecc0-b837-4dfb-9651-793b6f2986b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:16.546326 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:16.406572 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:34:16.696933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:16.628039 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:34:16.792881 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:16.726014 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": context deadline exceeded" Oct 31 19:34:16.823939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:16.730429 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faeecdd8-67fc-4249-ba29-ab06d605de57-kube-api-access-mgvwd" pod "faeecdd8-67fc-4249-ba29-ab06d605de57" (UID: "faeecdd8-67fc-4249-ba29-ab06d605de57"). InnerVolumeSpecName "kube-api-access-mgvwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:34:16.823939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:16.730565 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:16.953941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:16.948281 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:17.190485 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:17.190444 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgvwd\" (UniqueName: \"kubernetes.io/projected/faeecdd8-67fc-4249-ba29-ab06d605de57-kube-api-access-mgvwd\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:17.195000 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:17.194967 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faeecdd8-67fc-4249-ba29-ab06d605de57-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:17.278651 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:17.278606 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:17.351935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:17.351862 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:17.366812 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:17.366767 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:17.367391 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:17.367362 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:17.370008 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:17.369979 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:17.508779 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:17.506005 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:17.533009 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:17.532962 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:34:17.560166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:17.553218 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:34:17.560166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:17.557043 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:34:17.592095 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:17.575715 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:34:17.660938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:17.654749 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:17.699051 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:17.664042 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faeecdd8-67fc-4249-ba29-ab06d605de57-maas-api-xks-ca" pod "faeecdd8-67fc-4249-ba29-ab06d605de57" (UID: "faeecdd8-67fc-4249-ba29-ab06d605de57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:34:17.699051 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:34:17.681151 4910 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.928s" Oct 31 19:34:17.699051 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:17.681239 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:34:17.699051 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:17.681387 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e847aa56-zzsnz"] Oct 31 19:34:17.746937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:17.722952 4910 cpu_manager.go:414] "containerMap: removing container" podUID="faeecdd8-67fc-4249-ba29-ab06d605de57" containerName="revoke-keys" Oct 31 19:34:17.746937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:17.722989 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faeecdd8-67fc-4249-ba29-ab06d605de57" containerName="revoke-keys" Oct 31 19:34:17.746937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:17.723243 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="faeecdd8-67fc-4249-ba29-ab06d605de57" containerName="revoke-keys" Oct 31 19:34:17.855293 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:17.855235 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:34:17.871931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:17.856076 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:18.016930 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.016520 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:18.021833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:17.965816 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e847aa56-zzsnz" Oct 31 19:34:18.060530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:17.892149 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faeecdd8-67fc-4249-ba29-ab06d605de57-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:18.176964 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.176901 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d508d5-jk8c5" Oct 31 19:34:18.188742 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.188548 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4b24c82-9dba-4b45-994c-dbe26498219d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e847aa56-zzsnz\" (UID: \"e4b24c82-9dba-4b45-994c-dbe26498219d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e847aa56-zzsnz" Oct 31 19:34:18.188742 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.188605 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4b24c82-9dba-4b45-994c-dbe26498219d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e847aa56-zzsnz\" (UID: \"e4b24c82-9dba-4b45-994c-dbe26498219d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e847aa56-zzsnz" Oct 31 19:34:18.188742 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.188709 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz5zj\" (UniqueName: \"kubernetes.io/projected/e4b24c82-9dba-4b45-994c-dbe26498219d-kube-api-access-kz5zj\") pod \"maas-api-revoke-sub-e847aa56-zzsnz\" (UID: \"e4b24c82-9dba-4b45-994c-dbe26498219d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e847aa56-zzsnz" Oct 31 19:34:18.194951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.194381 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d508d5-jk8c5" event={"ID":"faeecdd8-67fc-4249-ba29-ab06d605de57","Type":"ContainerDied","Data":"0f82e46d7d463b5832307c0db523b1f8f04dfd69c0cbc4d916233c41c38e08c1"} Oct 31 19:34:18.194951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.194416 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f82e46d7d463b5832307c0db523b1f8f04dfd69c0cbc4d916233c41c38e08c1" Oct 31 19:34:18.194951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.194436 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e847aa56-zzsnz"] Oct 31 19:34:18.194951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.194484 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:34:18.194951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.194507 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:34:18.194951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.194544 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:34:18.194951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.194566 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7011-23b52b"] Oct 31 19:34:18.194951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.194606 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:34:18.194951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.194622 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7011-23b52b" event={"ID":"a2913723-bd1a-4797-af4f-c726356e2ed1","Type":"ContainerStarted","Data":"78dbaad815a058183ec21b1e986056073d114dc291b8a970630c8f2a533980e3"} Oct 31 19:34:18.194951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.194657 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:34:18.194951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.194690 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:34:18.194951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.194722 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:34:18.194951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.194751 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:34:18.194951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.194776 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:34:18.194951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.194797 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:34:18.194951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.194817 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:34:18.194951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.194838 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:34:18.194951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.194858 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:34:18.248445 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.248071 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aa3d18b-2tr54"] Oct 31 19:34:18.255264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.254191 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa3d18b-2tr54" Oct 31 19:34:18.256339 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.256281 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aa3d18b-2tr54"] Oct 31 19:34:18.292437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.292404 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfdfc824-4fbb-4053-ada5-1ab32efa036b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aa3d18b-2tr54\" (UID: \"dfdfc824-4fbb-4053-ada5-1ab32efa036b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa3d18b-2tr54" Oct 31 19:34:18.293019 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.292996 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfdfc824-4fbb-4053-ada5-1ab32efa036b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aa3d18b-2tr54\" (UID: \"dfdfc824-4fbb-4053-ada5-1ab32efa036b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa3d18b-2tr54" Oct 31 19:34:18.293187 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.293173 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4b24c82-9dba-4b45-994c-dbe26498219d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e847aa56-zzsnz\" (UID: \"e4b24c82-9dba-4b45-994c-dbe26498219d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e847aa56-zzsnz" Oct 31 19:34:18.293447 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.293422 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4b24c82-9dba-4b45-994c-dbe26498219d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e847aa56-zzsnz\" (UID: \"e4b24c82-9dba-4b45-994c-dbe26498219d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e847aa56-zzsnz" Oct 31 19:34:18.293519 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.293503 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqg9c\" (UniqueName: \"kubernetes.io/projected/dfdfc824-4fbb-4053-ada5-1ab32efa036b-kube-api-access-vqg9c\") pod \"maas-api-revoke-sub-3aa3d18b-2tr54\" (UID: \"dfdfc824-4fbb-4053-ada5-1ab32efa036b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa3d18b-2tr54" Oct 31 19:34:18.293713 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.293698 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz5zj\" (UniqueName: \"kubernetes.io/projected/e4b24c82-9dba-4b45-994c-dbe26498219d-kube-api-access-kz5zj\") pod \"maas-api-revoke-sub-e847aa56-zzsnz\" (UID: \"e4b24c82-9dba-4b45-994c-dbe26498219d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e847aa56-zzsnz" Oct 31 19:34:18.296412 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.296382 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4b24c82-9dba-4b45-994c-dbe26498219d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e847aa56-zzsnz\" (UID: \"e4b24c82-9dba-4b45-994c-dbe26498219d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e847aa56-zzsnz" Oct 31 19:34:18.307416 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.307384 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4b24c82-9dba-4b45-994c-dbe26498219d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e847aa56-zzsnz\" (UID: \"e4b24c82-9dba-4b45-994c-dbe26498219d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e847aa56-zzsnz" Oct 31 19:34:18.318296 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.318273 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz5zj\" (UniqueName: \"kubernetes.io/projected/e4b24c82-9dba-4b45-994c-dbe26498219d-kube-api-access-kz5zj\") pod \"maas-api-revoke-sub-e847aa56-zzsnz\" (UID: \"e4b24c82-9dba-4b45-994c-dbe26498219d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e847aa56-zzsnz" Oct 31 19:34:18.398348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.397226 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfdfc824-4fbb-4053-ada5-1ab32efa036b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aa3d18b-2tr54\" (UID: \"dfdfc824-4fbb-4053-ada5-1ab32efa036b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa3d18b-2tr54" Oct 31 19:34:18.398348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.397283 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfdfc824-4fbb-4053-ada5-1ab32efa036b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aa3d18b-2tr54\" (UID: \"dfdfc824-4fbb-4053-ada5-1ab32efa036b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa3d18b-2tr54" Oct 31 19:34:18.398348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.397386 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqg9c\" (UniqueName: \"kubernetes.io/projected/dfdfc824-4fbb-4053-ada5-1ab32efa036b-kube-api-access-vqg9c\") pod \"maas-api-revoke-sub-3aa3d18b-2tr54\" (UID: \"dfdfc824-4fbb-4053-ada5-1ab32efa036b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa3d18b-2tr54" Oct 31 19:34:18.399213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.399175 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfdfc824-4fbb-4053-ada5-1ab32efa036b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aa3d18b-2tr54\" (UID: \"dfdfc824-4fbb-4053-ada5-1ab32efa036b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa3d18b-2tr54" Oct 31 19:34:18.404649 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.404616 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfdfc824-4fbb-4053-ada5-1ab32efa036b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aa3d18b-2tr54\" (UID: \"dfdfc824-4fbb-4053-ada5-1ab32efa036b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa3d18b-2tr54" Oct 31 19:34:18.415946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.415899 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqg9c\" (UniqueName: \"kubernetes.io/projected/dfdfc824-4fbb-4053-ada5-1ab32efa036b-kube-api-access-vqg9c\") pod \"maas-api-revoke-sub-3aa3d18b-2tr54\" (UID: \"dfdfc824-4fbb-4053-ada5-1ab32efa036b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa3d18b-2tr54" Oct 31 19:34:18.485707 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.485360 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e847aa56-zzsnz" Oct 31 19:34:18.583975 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.574775 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa3d18b-2tr54" Oct 31 19:34:18.881678 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.881643 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e847aa56-zzsnz"] Oct 31 19:34:18.964115 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:18.964086 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aa3d18b-2tr54"] Oct 31 19:34:19.199386 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:19.199301 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e847aa56-zzsnz" event={"ID":"e4b24c82-9dba-4b45-994c-dbe26498219d","Type":"ContainerStarted","Data":"13ed584a77769d85b76802e56477789b9cdb720239c27f116eb700bf53cd835a"} Oct 31 19:34:19.202108 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:19.202050 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa3d18b-2tr54" event={"ID":"dfdfc824-4fbb-4053-ada5-1ab32efa036b","Type":"ContainerStarted","Data":"47c0616fc270962c175a00b3c809dcf9104992649e8cdd5ea076abfac809475e"} Oct 31 19:34:20.215157 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:20.215116 4910 generic.go:356] "Generic (PLEG): container finished" podID="e4b24c82-9dba-4b45-994c-dbe26498219d" containerID="ba0aad91581763dfba6e46119cdfc33ba2b5ddad4751e7e86176400a1ee0ccdd" exitCode=0 Oct 31 19:34:20.215810 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:20.215782 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e847aa56-zzsnz" event={"ID":"e4b24c82-9dba-4b45-994c-dbe26498219d","Type":"ContainerDied","Data":"ba0aad91581763dfba6e46119cdfc33ba2b5ddad4751e7e86176400a1ee0ccdd"} Oct 31 19:34:20.217969 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:20.217896 4910 generic.go:356] "Generic (PLEG): container finished" podID="dfdfc824-4fbb-4053-ada5-1ab32efa036b" containerID="16b53801741ca46062ba8201305dcc0c6415468863f5e109a9e89d6cdc9857d9" exitCode=0 Oct 31 19:34:20.217969 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:20.217946 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa3d18b-2tr54" event={"ID":"dfdfc824-4fbb-4053-ada5-1ab32efa036b","Type":"ContainerDied","Data":"16b53801741ca46062ba8201305dcc0c6415468863f5e109a9e89d6cdc9857d9"} Oct 31 19:34:21.298504 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:21.298429 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7011-23b52b" event={"ID":"a2913723-bd1a-4797-af4f-c726356e2ed1","Type":"ContainerStarted","Data":"516a7de705040e40f5fcb5c78197cce3928797987c5b95ec427489bbc179036c"} Oct 31 19:34:21.311931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:21.306534 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7011-23b52b" podStartSLOduration=5.960895927 podStartE2EDuration="11.306513005s" totalImagesPullingTime="5.345617078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:21.300991924 +0000 UTC m=+895.663812297" watchObservedRunningTime="2026-10-31 19:34:21.306513005 +0000 UTC m=+895.669333378" Oct 31 19:34:21.586801 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:21.586741 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa3d18b-2tr54" Oct 31 19:34:21.626633 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:21.626608 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e847aa56-zzsnz" Oct 31 19:34:21.766486 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:21.766446 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfdfc824-4fbb-4053-ada5-1ab32efa036b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfdfc824-4fbb-4053-ada5-1ab32efa036b-maas-api-openshift-ca\") pod \"dfdfc824-4fbb-4053-ada5-1ab32efa036b\" (UID: \"dfdfc824-4fbb-4053-ada5-1ab32efa036b\") " Oct 31 19:34:21.766678 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:21.766577 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4b24c82-9dba-4b45-994c-dbe26498219d-kube-api-access-kz5zj\" (UniqueName: \"kubernetes.io/projected/e4b24c82-9dba-4b45-994c-dbe26498219d-kube-api-access-kz5zj\") pod \"e4b24c82-9dba-4b45-994c-dbe26498219d\" (UID: \"e4b24c82-9dba-4b45-994c-dbe26498219d\") " Oct 31 19:34:21.766678 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:21.766637 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfdfc824-4fbb-4053-ada5-1ab32efa036b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfdfc824-4fbb-4053-ada5-1ab32efa036b-maas-api-xks-ca\") pod \"dfdfc824-4fbb-4053-ada5-1ab32efa036b\" (UID: \"dfdfc824-4fbb-4053-ada5-1ab32efa036b\") " Oct 31 19:34:21.766791 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:21.766696 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4b24c82-9dba-4b45-994c-dbe26498219d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4b24c82-9dba-4b45-994c-dbe26498219d-maas-api-openshift-ca\") pod \"e4b24c82-9dba-4b45-994c-dbe26498219d\" (UID: \"e4b24c82-9dba-4b45-994c-dbe26498219d\") " Oct 31 19:34:21.766791 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:21.766741 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfdfc824-4fbb-4053-ada5-1ab32efa036b-kube-api-access-vqg9c\" (UniqueName: \"kubernetes.io/projected/dfdfc824-4fbb-4053-ada5-1ab32efa036b-kube-api-access-vqg9c\") pod \"dfdfc824-4fbb-4053-ada5-1ab32efa036b\" (UID: \"dfdfc824-4fbb-4053-ada5-1ab32efa036b\") " Oct 31 19:34:21.767073 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:21.767050 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4b24c82-9dba-4b45-994c-dbe26498219d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4b24c82-9dba-4b45-994c-dbe26498219d-maas-api-xks-ca\") pod \"e4b24c82-9dba-4b45-994c-dbe26498219d\" (UID: \"e4b24c82-9dba-4b45-994c-dbe26498219d\") " Oct 31 19:34:21.767257 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:21.767199 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfdfc824-4fbb-4053-ada5-1ab32efa036b-maas-api-openshift-ca" pod "dfdfc824-4fbb-4053-ada5-1ab32efa036b" (UID: "dfdfc824-4fbb-4053-ada5-1ab32efa036b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:34:21.767958 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:21.767881 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b24c82-9dba-4b45-994c-dbe26498219d-maas-api-openshift-ca" pod "e4b24c82-9dba-4b45-994c-dbe26498219d" (UID: "e4b24c82-9dba-4b45-994c-dbe26498219d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:34:21.767958 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:21.767895 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfdfc824-4fbb-4053-ada5-1ab32efa036b-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:21.769123 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:21.769100 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b24c82-9dba-4b45-994c-dbe26498219d-kube-api-access-kz5zj" pod "e4b24c82-9dba-4b45-994c-dbe26498219d" (UID: "e4b24c82-9dba-4b45-994c-dbe26498219d"). InnerVolumeSpecName "kube-api-access-kz5zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:34:21.769352 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:21.769299 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfdfc824-4fbb-4053-ada5-1ab32efa036b-maas-api-xks-ca" pod "dfdfc824-4fbb-4053-ada5-1ab32efa036b" (UID: "dfdfc824-4fbb-4053-ada5-1ab32efa036b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:34:21.770299 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:21.770273 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfdfc824-4fbb-4053-ada5-1ab32efa036b-kube-api-access-vqg9c" pod "dfdfc824-4fbb-4053-ada5-1ab32efa036b" (UID: "dfdfc824-4fbb-4053-ada5-1ab32efa036b"). InnerVolumeSpecName "kube-api-access-vqg9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:34:21.770984 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:21.770961 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4b24c82-9dba-4b45-994c-dbe26498219d-maas-api-xks-ca" pod "e4b24c82-9dba-4b45-994c-dbe26498219d" (UID: "e4b24c82-9dba-4b45-994c-dbe26498219d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:34:21.869407 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:21.869325 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4b24c82-9dba-4b45-994c-dbe26498219d-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:21.869407 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:21.869357 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz5zj\" (UniqueName: \"kubernetes.io/projected/e4b24c82-9dba-4b45-994c-dbe26498219d-kube-api-access-kz5zj\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:21.869407 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:21.869373 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfdfc824-4fbb-4053-ada5-1ab32efa036b-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:21.869407 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:21.869387 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4b24c82-9dba-4b45-994c-dbe26498219d-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:21.869407 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:21.869397 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqg9c\" (UniqueName: \"kubernetes.io/projected/dfdfc824-4fbb-4053-ada5-1ab32efa036b-kube-api-access-vqg9c\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:22.111593 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:22.111545 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7011-23b52b"] Oct 31 19:34:22.135722 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:22.135620 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7011-23b52b"] Oct 31 19:34:22.296420 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:22.296336 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa3d18b-2tr54" event={"ID":"dfdfc824-4fbb-4053-ada5-1ab32efa036b","Type":"ContainerDied","Data":"47c0616fc270962c175a00b3c809dcf9104992649e8cdd5ea076abfac809475e"} Oct 31 19:34:22.296695 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:22.296472 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47c0616fc270962c175a00b3c809dcf9104992649e8cdd5ea076abfac809475e" Oct 31 19:34:22.296695 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:22.296572 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa3d18b-2tr54" Oct 31 19:34:22.309776 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:22.305673 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e847aa56-zzsnz" Oct 31 19:34:22.309776 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:22.307503 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e847aa56-zzsnz" event={"ID":"e4b24c82-9dba-4b45-994c-dbe26498219d","Type":"ContainerDied","Data":"13ed584a77769d85b76802e56477789b9cdb720239c27f116eb700bf53cd835a"} Oct 31 19:34:22.309776 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:22.307567 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13ed584a77769d85b76802e56477789b9cdb720239c27f116eb700bf53cd835a" Oct 31 19:34:23.327343 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:23.327190 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7011-23b52b" podUID="a2913723-bd1a-4797-af4f-c726356e2ed1" containerName="test-curl-7011-23b52b" containerID="cri-o://516a7de705040e40f5fcb5c78197cce3928797987c5b95ec427489bbc179036c" gracePeriod=2 Oct 31 19:34:23.333953 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:34:23.330400 4910 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7011-23b52b\" is forbidden: User \"system:node:ip-10-0-142-116.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-116.us-east-2.compute.internal' and this object" podUID="a2913723-bd1a-4797-af4f-c726356e2ed1" pod="opendatahub/test-curl-7011-23b52b" Oct 31 19:34:24.813318 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:24.813241 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abe32a79-lgvdb"] Oct 31 19:34:24.814497 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:24.814321 4910 cpu_manager.go:414] "containerMap: removing container" podUID="dfdfc824-4fbb-4053-ada5-1ab32efa036b" containerName="revoke-keys" Oct 31 19:34:24.814497 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:24.814346 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfdfc824-4fbb-4053-ada5-1ab32efa036b" containerName="revoke-keys" Oct 31 19:34:24.814497 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:24.814381 4910 cpu_manager.go:414] "containerMap: removing container" podUID="a2913723-bd1a-4797-af4f-c726356e2ed1" containerName="test-curl-7011-23b52b" Oct 31 19:34:24.814497 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:24.814388 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2913723-bd1a-4797-af4f-c726356e2ed1" containerName="test-curl-7011-23b52b" Oct 31 19:34:24.814497 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:24.814448 4910 cpu_manager.go:414] "containerMap: removing container" podUID="e4b24c82-9dba-4b45-994c-dbe26498219d" containerName="revoke-keys" Oct 31 19:34:24.814497 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:24.814456 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4b24c82-9dba-4b45-994c-dbe26498219d" containerName="revoke-keys" Oct 31 19:34:24.814900 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:24.814699 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2913723-bd1a-4797-af4f-c726356e2ed1" containerName="test-curl-7011-23b52b" Oct 31 19:34:24.814900 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:24.814730 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4b24c82-9dba-4b45-994c-dbe26498219d" containerName="revoke-keys" Oct 31 19:34:24.814900 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:24.814742 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfdfc824-4fbb-4053-ada5-1ab32efa036b" containerName="revoke-keys" Oct 31 19:34:24.823493 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:24.823467 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe32a79-lgvdb" Oct 31 19:34:24.834691 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:24.827897 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:34:24.834691 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:24.830004 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:34:24.853628 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:24.853594 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abe32a79-lgvdb"] Oct 31 19:34:24.943406 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:24.943367 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chhv6\" (UniqueName: \"kubernetes.io/projected/ebb4ece9-4a96-4027-969a-6a4884f5205b-kube-api-access-chhv6\") pod \"maas-api-revoke-sub-abe32a79-lgvdb\" (UID: \"ebb4ece9-4a96-4027-969a-6a4884f5205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe32a79-lgvdb" Oct 31 19:34:24.943625 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:24.943599 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb4ece9-4a96-4027-969a-6a4884f5205b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abe32a79-lgvdb\" (UID: \"ebb4ece9-4a96-4027-969a-6a4884f5205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe32a79-lgvdb" Oct 31 19:34:24.943784 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:24.943768 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb4ece9-4a96-4027-969a-6a4884f5205b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abe32a79-lgvdb\" (UID: \"ebb4ece9-4a96-4027-969a-6a4884f5205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe32a79-lgvdb" Oct 31 19:34:25.053486 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:25.052974 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb4ece9-4a96-4027-969a-6a4884f5205b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abe32a79-lgvdb\" (UID: \"ebb4ece9-4a96-4027-969a-6a4884f5205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe32a79-lgvdb" Oct 31 19:34:25.053486 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:25.053127 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb4ece9-4a96-4027-969a-6a4884f5205b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abe32a79-lgvdb\" (UID: \"ebb4ece9-4a96-4027-969a-6a4884f5205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe32a79-lgvdb" Oct 31 19:34:25.053486 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:25.053307 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chhv6\" (UniqueName: \"kubernetes.io/projected/ebb4ece9-4a96-4027-969a-6a4884f5205b-kube-api-access-chhv6\") pod \"maas-api-revoke-sub-abe32a79-lgvdb\" (UID: \"ebb4ece9-4a96-4027-969a-6a4884f5205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe32a79-lgvdb" Oct 31 19:34:25.055204 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:25.055131 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb4ece9-4a96-4027-969a-6a4884f5205b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abe32a79-lgvdb\" (UID: \"ebb4ece9-4a96-4027-969a-6a4884f5205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe32a79-lgvdb" Oct 31 19:34:25.058450 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:25.058426 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb4ece9-4a96-4027-969a-6a4884f5205b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abe32a79-lgvdb\" (UID: \"ebb4ece9-4a96-4027-969a-6a4884f5205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe32a79-lgvdb" Oct 31 19:34:25.067545 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:25.067489 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chhv6\" (UniqueName: \"kubernetes.io/projected/ebb4ece9-4a96-4027-969a-6a4884f5205b-kube-api-access-chhv6\") pod \"maas-api-revoke-sub-abe32a79-lgvdb\" (UID: \"ebb4ece9-4a96-4027-969a-6a4884f5205b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe32a79-lgvdb" Oct 31 19:34:25.154035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:25.153988 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe32a79-lgvdb" Oct 31 19:34:25.452801 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:34:25.452770 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb4ece9_4a96_4027_969a_6a4884f5205b.slice/crio-2d2f193842762320b187cd4cfa3a076a0006bbee323c07ecfc6a3cc4223771e2 WatchSource:0}: Error finding container 2d2f193842762320b187cd4cfa3a076a0006bbee323c07ecfc6a3cc4223771e2: Status 404 returned error can't find the container with id 2d2f193842762320b187cd4cfa3a076a0006bbee323c07ecfc6a3cc4223771e2 Oct 31 19:34:25.495326 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:25.495260 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abe32a79-lgvdb"] Oct 31 19:34:25.631883 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:25.631507 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-23b52b" Oct 31 19:34:25.676272 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:25.676243 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2913723-bd1a-4797-af4f-c726356e2ed1-kube-api-access-p56tl\" (UniqueName: \"kubernetes.io/projected/a2913723-bd1a-4797-af4f-c726356e2ed1-kube-api-access-p56tl\") pod \"a2913723-bd1a-4797-af4f-c726356e2ed1\" (UID: \"a2913723-bd1a-4797-af4f-c726356e2ed1\") " Oct 31 19:34:25.678326 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:25.678299 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2913723-bd1a-4797-af4f-c726356e2ed1-kube-api-access-p56tl" pod "a2913723-bd1a-4797-af4f-c726356e2ed1" (UID: "a2913723-bd1a-4797-af4f-c726356e2ed1"). InnerVolumeSpecName "kube-api-access-p56tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:34:25.757141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:25.757103 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2913723-bd1a-4797-af4f-c726356e2ed1" path="/var/lib/kubelet/pods/a2913723-bd1a-4797-af4f-c726356e2ed1/volumes" Oct 31 19:34:25.779154 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:25.779122 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p56tl\" (UniqueName: \"kubernetes.io/projected/a2913723-bd1a-4797-af4f-c726356e2ed1-kube-api-access-p56tl\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:25.999726 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:25.999693 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 19:34:26.000645 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:26.000624 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 19:34:26.361066 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:26.361028 4910 generic.go:356] "Generic (PLEG): container finished" podID="a2913723-bd1a-4797-af4f-c726356e2ed1" containerID="516a7de705040e40f5fcb5c78197cce3928797987c5b95ec427489bbc179036c" exitCode=137 Oct 31 19:34:26.361262 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:26.361086 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-23b52b" Oct 31 19:34:26.361262 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:26.361143 4910 scope.go:122] "RemoveContainer" containerID="516a7de705040e40f5fcb5c78197cce3928797987c5b95ec427489bbc179036c" Oct 31 19:34:26.363344 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:26.363305 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe32a79-lgvdb" event={"ID":"ebb4ece9-4a96-4027-969a-6a4884f5205b","Type":"ContainerStarted","Data":"3abdce930a26dbb5aadcee505155eec753702274e944d78d01bedf882d6ebeeb"} Oct 31 19:34:26.363467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:26.363371 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe32a79-lgvdb" event={"ID":"ebb4ece9-4a96-4027-969a-6a4884f5205b","Type":"ContainerStarted","Data":"2d2f193842762320b187cd4cfa3a076a0006bbee323c07ecfc6a3cc4223771e2"} Oct 31 19:34:26.372942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:26.372906 4910 scope.go:122] "RemoveContainer" containerID="516a7de705040e40f5fcb5c78197cce3928797987c5b95ec427489bbc179036c" Oct 31 19:34:26.373267 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:34:26.373245 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"516a7de705040e40f5fcb5c78197cce3928797987c5b95ec427489bbc179036c\": container with ID starting with 516a7de705040e40f5fcb5c78197cce3928797987c5b95ec427489bbc179036c not found: ID does not exist" containerID="516a7de705040e40f5fcb5c78197cce3928797987c5b95ec427489bbc179036c" Oct 31 19:34:26.373359 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:26.373276 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"516a7de705040e40f5fcb5c78197cce3928797987c5b95ec427489bbc179036c"} err="failed to get container status \"516a7de705040e40f5fcb5c78197cce3928797987c5b95ec427489bbc179036c\": rpc error: code = NotFound desc = could not find container \"516a7de705040e40f5fcb5c78197cce3928797987c5b95ec427489bbc179036c\": container with ID starting with 516a7de705040e40f5fcb5c78197cce3928797987c5b95ec427489bbc179036c not found: ID does not exist" Oct 31 19:34:26.394739 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:26.394692 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe32a79-lgvdb" podStartSLOduration=2.394679546 podStartE2EDuration="2.394679546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:26.389348253 +0000 UTC m=+900.752168629" watchObservedRunningTime="2026-10-31 19:34:26.394679546 +0000 UTC m=+900.757499917" Oct 31 19:34:26.823294 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:26.814076 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh"] Oct 31 19:34:26.823294 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:26.820934 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:26.826250 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:26.826218 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh"] Oct 31 19:34:26.828959 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:26.827268 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f0303f0d-openshift-default-dockercfg-kvt47\"" Oct 31 19:34:26.898191 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:26.898095 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfcsn\" (UniqueName: \"kubernetes.io/projected/26fe161a-441f-418c-8c6b-1b108999ca7d-kube-api-access-gfcsn\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:26.898354 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:26.898225 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-workload-socket\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:26.898354 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:26.898258 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-data\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:26.898354 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:26.898304 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-podinfo\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:26.898559 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:26.898536 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26fe161a-441f-418c-8c6b-1b108999ca7d-istiod-ca-cert\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:26.898610 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:26.898572 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-credential-socket\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:26.898741 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:26.898705 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-envoy\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:26.898801 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:26.898742 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-workload-certs\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:26.898857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:26.898826 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-token\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:27.000406 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:27.000362 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-workload-certs\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:27.000872 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:27.000445 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-token\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:27.000872 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:27.000554 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfcsn\" (UniqueName: \"kubernetes.io/projected/26fe161a-441f-418c-8c6b-1b108999ca7d-kube-api-access-gfcsn\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:27.000872 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:27.000627 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-workload-socket\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:27.000872 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:27.000659 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-data\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:27.000872 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:27.000719 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-podinfo\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:27.000872 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:27.000821 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-workload-certs\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:27.001208 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:27.000965 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26fe161a-441f-418c-8c6b-1b108999ca7d-istiod-ca-cert\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:27.001208 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:27.001000 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-credential-socket\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:27.001208 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:27.001060 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-workload-socket\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:27.001208 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:27.001104 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-data\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:27.001208 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:27.001130 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-envoy\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:27.001474 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:27.001348 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-credential-socket\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:27.001726 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:27.001700 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26fe161a-441f-418c-8c6b-1b108999ca7d-istiod-ca-cert\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:27.003643 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:27.003621 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-envoy\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:27.004156 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:27.004126 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-podinfo\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:27.038719 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:27.016394 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfcsn\" (UniqueName: \"kubernetes.io/projected/26fe161a-441f-418c-8c6b-1b108999ca7d-kube-api-access-gfcsn\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:27.038719 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:27.018720 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-token\") pod \"e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:27.143467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:27.143426 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:27.671949 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:27.670144 4910 generic.go:356] "Generic (PLEG): container finished" podID="ebb4ece9-4a96-4027-969a-6a4884f5205b" containerID="3abdce930a26dbb5aadcee505155eec753702274e944d78d01bedf882d6ebeeb" exitCode=0 Oct 31 19:34:27.671949 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:27.670421 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe32a79-lgvdb" event={"ID":"ebb4ece9-4a96-4027-969a-6a4884f5205b","Type":"ContainerDied","Data":"3abdce930a26dbb5aadcee505155eec753702274e944d78d01bedf882d6ebeeb"} Oct 31 19:34:27.720589 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:27.718829 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:34:27.720589 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:27.718935 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:34:27.720589 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:27.718988 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:34:27.720589 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:27.720217 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh"] Oct 31 19:34:28.687947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:28.687274 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" event={"ID":"26fe161a-441f-418c-8c6b-1b108999ca7d","Type":"ContainerStarted","Data":"20d854492ed778f2a72d2b68b35e50f0fb78f37264fe0a23284f4928258e2743"} Oct 31 19:34:28.687947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:28.687394 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" event={"ID":"26fe161a-441f-418c-8c6b-1b108999ca7d","Type":"ContainerStarted","Data":"8715311ee63f844a75b2472354e036105d0525616862bd85beca04f35837d891"} Oct 31 19:34:28.728977 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:28.718169 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" podStartSLOduration=2.718146286 podStartE2EDuration="2.718146286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:28.712429537 +0000 UTC m=+903.075249909" watchObservedRunningTime="2026-10-31 19:34:28.718146286 +0000 UTC m=+903.080966765" Oct 31 19:34:28.930992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:28.927936 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe32a79-lgvdb" Oct 31 19:34:28.995521 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:28.995435 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebb4ece9-4a96-4027-969a-6a4884f5205b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb4ece9-4a96-4027-969a-6a4884f5205b-maas-api-xks-ca\") pod \"ebb4ece9-4a96-4027-969a-6a4884f5205b\" (UID: \"ebb4ece9-4a96-4027-969a-6a4884f5205b\") " Oct 31 19:34:28.995658 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:28.995527 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebb4ece9-4a96-4027-969a-6a4884f5205b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb4ece9-4a96-4027-969a-6a4884f5205b-maas-api-openshift-ca\") pod \"ebb4ece9-4a96-4027-969a-6a4884f5205b\" (UID: \"ebb4ece9-4a96-4027-969a-6a4884f5205b\") " Oct 31 19:34:28.995740 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:28.995722 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebb4ece9-4a96-4027-969a-6a4884f5205b-kube-api-access-chhv6\" (UniqueName: \"kubernetes.io/projected/ebb4ece9-4a96-4027-969a-6a4884f5205b-kube-api-access-chhv6\") pod \"ebb4ece9-4a96-4027-969a-6a4884f5205b\" (UID: \"ebb4ece9-4a96-4027-969a-6a4884f5205b\") " Oct 31 19:34:28.997316 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:28.997280 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb4ece9-4a96-4027-969a-6a4884f5205b-maas-api-openshift-ca" pod "ebb4ece9-4a96-4027-969a-6a4884f5205b" (UID: "ebb4ece9-4a96-4027-969a-6a4884f5205b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:34:29.003618 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:29.003565 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb4ece9-4a96-4027-969a-6a4884f5205b-kube-api-access-chhv6" pod "ebb4ece9-4a96-4027-969a-6a4884f5205b" (UID: "ebb4ece9-4a96-4027-969a-6a4884f5205b"). InnerVolumeSpecName "kube-api-access-chhv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:34:29.004792 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:29.004759 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebb4ece9-4a96-4027-969a-6a4884f5205b-maas-api-xks-ca" pod "ebb4ece9-4a96-4027-969a-6a4884f5205b" (UID: "ebb4ece9-4a96-4027-969a-6a4884f5205b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:34:29.099348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:29.099316 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebb4ece9-4a96-4027-969a-6a4884f5205b-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:29.099348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:29.099349 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebb4ece9-4a96-4027-969a-6a4884f5205b-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:29.099577 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:29.099365 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chhv6\" (UniqueName: \"kubernetes.io/projected/ebb4ece9-4a96-4027-969a-6a4884f5205b-kube-api-access-chhv6\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:29.144273 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:29.144242 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:29.150894 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:29.150871 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:29.697358 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:29.697324 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe32a79-lgvdb" Oct 31 19:34:29.697358 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:29.697345 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abe32a79-lgvdb" event={"ID":"ebb4ece9-4a96-4027-969a-6a4884f5205b","Type":"ContainerDied","Data":"2d2f193842762320b187cd4cfa3a076a0006bbee323c07ecfc6a3cc4223771e2"} Oct 31 19:34:29.697836 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:29.697408 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d2f193842762320b187cd4cfa3a076a0006bbee323c07ecfc6a3cc4223771e2" Oct 31 19:34:29.697836 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:29.697748 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:29.698767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:29.698742 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:34:30.857192 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:30.857150 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7"] Oct 31 19:34:30.858997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:30.858867 4910 cpu_manager.go:414] "containerMap: removing container" podUID="ebb4ece9-4a96-4027-969a-6a4884f5205b" containerName="revoke-keys" Oct 31 19:34:30.859183 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:30.859155 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebb4ece9-4a96-4027-969a-6a4884f5205b" containerName="revoke-keys" Oct 31 19:34:30.859646 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:30.859615 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebb4ece9-4a96-4027-969a-6a4884f5205b" containerName="revoke-keys" Oct 31 19:34:30.882407 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:30.882344 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:30.894455 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:30.894412 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1cc6560c-openshift-default-dockercfg-2nvcl\"" Oct 31 19:34:30.999019 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:30.998953 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7"] Oct 31 19:34:31.103007 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.102964 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istiod-ca-cert\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.103199 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.103048 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-data\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.103199 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.103101 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj48w\" (UniqueName: \"kubernetes.io/projected/099d2ac1-f9dc-4b27-b75e-b13b45b22148-kube-api-access-bj48w\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.103199 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.103136 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-envoy\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.103462 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.103267 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-token\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.103462 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.103389 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-podinfo\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.107319 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.103665 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-workload-certs\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.107319 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.103706 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-credential-socket\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.107319 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.103757 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-workload-socket\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.205441 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.205401 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-workload-certs\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.205623 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.205536 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-credential-socket\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.205623 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.205588 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-workload-socket\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.205740 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.205686 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istiod-ca-cert\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.205795 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.205747 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-data\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.205859 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.205793 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj48w\" (UniqueName: \"kubernetes.io/projected/099d2ac1-f9dc-4b27-b75e-b13b45b22148-kube-api-access-bj48w\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.205859 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.205834 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-workload-certs\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.205989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.205895 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-credential-socket\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.206200 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.205825 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-envoy\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.206436 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.206409 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istiod-ca-cert\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.206559 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.206478 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-token\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.206559 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.206296 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-workload-socket\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.206559 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.206187 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-data\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.206724 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.206604 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-podinfo\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.208235 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.208211 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-envoy\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.209440 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.209420 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-podinfo\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.218795 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.218768 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj48w\" (UniqueName: \"kubernetes.io/projected/099d2ac1-f9dc-4b27-b75e-b13b45b22148-kube-api-access-bj48w\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.219044 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.219018 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-token\") pod \"e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.304824 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.304782 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:31.612195 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:34:31.612159 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099d2ac1_f9dc_4b27_b75e_b13b45b22148.slice/crio-15f20b2e0a6abf46ddca65b943590fe942c32cdbfd50c21e957e0971e1feb72a WatchSource:0}: Error finding container 15f20b2e0a6abf46ddca65b943590fe942c32cdbfd50c21e957e0971e1feb72a: Status 404 returned error can't find the container with id 15f20b2e0a6abf46ddca65b943590fe942c32cdbfd50c21e957e0971e1feb72a Oct 31 19:34:31.615159 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.615133 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7"] Oct 31 19:34:31.615274 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.615217 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:34:31.615333 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.615275 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:34:31.615333 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.615303 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:34:31.789636 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:31.789595 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" event={"ID":"099d2ac1-f9dc-4b27-b75e-b13b45b22148","Type":"ContainerStarted","Data":"15f20b2e0a6abf46ddca65b943590fe942c32cdbfd50c21e957e0971e1feb72a"} Oct 31 19:34:32.800267 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:32.800228 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" event={"ID":"099d2ac1-f9dc-4b27-b75e-b13b45b22148","Type":"ContainerStarted","Data":"59c5bc1fd06f676f1098a0dcfc8f5547d39af133abff507c8e549ec3ae90f231"} Oct 31 19:34:32.828831 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:32.828777 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" podStartSLOduration=2.828763087 podStartE2EDuration="2.828763087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:32.82508452 +0000 UTC m=+907.187904892" watchObservedRunningTime="2026-10-31 19:34:32.828763087 +0000 UTC m=+907.191583459" Oct 31 19:34:33.305690 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:33.305656 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:33.311189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:33.311165 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:33.809713 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:33.809680 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:33.810588 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:33.810567 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:34:39.607185 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:39.607146 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de1f4432-5xcqs"] Oct 31 19:34:39.615366 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:39.615340 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de1f4432-5xcqs" Oct 31 19:34:39.620526 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:39.619738 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:34:39.620526 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:39.619755 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:34:39.620526 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:39.620418 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de1f4432-5xcqs"] Oct 31 19:34:39.689025 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:39.688989 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br75p\" (UniqueName: \"kubernetes.io/projected/bc81cb01-6336-47e0-8d44-466ccaa83704-kube-api-access-br75p\") pod \"maas-api-revoke-sub-de1f4432-5xcqs\" (UID: \"bc81cb01-6336-47e0-8d44-466ccaa83704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de1f4432-5xcqs" Oct 31 19:34:39.689204 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:39.689059 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc81cb01-6336-47e0-8d44-466ccaa83704-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de1f4432-5xcqs\" (UID: \"bc81cb01-6336-47e0-8d44-466ccaa83704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de1f4432-5xcqs" Oct 31 19:34:39.689332 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:39.689309 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc81cb01-6336-47e0-8d44-466ccaa83704-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de1f4432-5xcqs\" (UID: \"bc81cb01-6336-47e0-8d44-466ccaa83704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de1f4432-5xcqs" Oct 31 19:34:39.790769 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:39.790725 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br75p\" (UniqueName: \"kubernetes.io/projected/bc81cb01-6336-47e0-8d44-466ccaa83704-kube-api-access-br75p\") pod \"maas-api-revoke-sub-de1f4432-5xcqs\" (UID: \"bc81cb01-6336-47e0-8d44-466ccaa83704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de1f4432-5xcqs" Oct 31 19:34:39.790956 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:39.790837 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc81cb01-6336-47e0-8d44-466ccaa83704-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de1f4432-5xcqs\" (UID: \"bc81cb01-6336-47e0-8d44-466ccaa83704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de1f4432-5xcqs" Oct 31 19:34:39.791079 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:39.791059 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc81cb01-6336-47e0-8d44-466ccaa83704-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de1f4432-5xcqs\" (UID: \"bc81cb01-6336-47e0-8d44-466ccaa83704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de1f4432-5xcqs" Oct 31 19:34:39.791590 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:39.791556 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc81cb01-6336-47e0-8d44-466ccaa83704-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de1f4432-5xcqs\" (UID: \"bc81cb01-6336-47e0-8d44-466ccaa83704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de1f4432-5xcqs" Oct 31 19:34:39.793517 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:39.793495 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc81cb01-6336-47e0-8d44-466ccaa83704-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de1f4432-5xcqs\" (UID: \"bc81cb01-6336-47e0-8d44-466ccaa83704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de1f4432-5xcqs" Oct 31 19:34:39.809086 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:39.809058 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br75p\" (UniqueName: \"kubernetes.io/projected/bc81cb01-6336-47e0-8d44-466ccaa83704-kube-api-access-br75p\") pod \"maas-api-revoke-sub-de1f4432-5xcqs\" (UID: \"bc81cb01-6336-47e0-8d44-466ccaa83704\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de1f4432-5xcqs" Oct 31 19:34:39.938073 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:39.937992 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de1f4432-5xcqs" Oct 31 19:34:40.232429 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:34:40.232309 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc81cb01_6336_47e0_8d44_466ccaa83704.slice/crio-9da1742eec16869516f0d309b6e5ef6b6b06bcc0320d89ed9282013fe768cc6f WatchSource:0}: Error finding container 9da1742eec16869516f0d309b6e5ef6b6b06bcc0320d89ed9282013fe768cc6f: Status 404 returned error can't find the container with id 9da1742eec16869516f0d309b6e5ef6b6b06bcc0320d89ed9282013fe768cc6f Oct 31 19:34:40.233876 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:40.233852 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de1f4432-5xcqs"] Oct 31 19:34:40.253973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:40.253945 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7023-ea6cff"] Oct 31 19:34:40.261600 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:40.261572 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7023-ea6cff" Oct 31 19:34:40.266286 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:40.266258 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7023-ea6cff"] Oct 31 19:34:40.272784 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:40.272723 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4pmld\"" Oct 31 19:34:40.302965 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:40.301987 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q8kv\" (UniqueName: \"kubernetes.io/projected/2536d01a-00ee-4480-8462-3f863cbfca13-kube-api-access-2q8kv\") pod \"test-curl-7023-ea6cff\" (UID: \"2536d01a-00ee-4480-8462-3f863cbfca13\") " pod="opendatahub/test-curl-7023-ea6cff" Oct 31 19:34:40.404126 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:40.404084 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q8kv\" (UniqueName: \"kubernetes.io/projected/2536d01a-00ee-4480-8462-3f863cbfca13-kube-api-access-2q8kv\") pod \"test-curl-7023-ea6cff\" (UID: \"2536d01a-00ee-4480-8462-3f863cbfca13\") " pod="opendatahub/test-curl-7023-ea6cff" Oct 31 19:34:40.419118 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:40.419083 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q8kv\" (UniqueName: \"kubernetes.io/projected/2536d01a-00ee-4480-8462-3f863cbfca13-kube-api-access-2q8kv\") pod \"test-curl-7023-ea6cff\" (UID: \"2536d01a-00ee-4480-8462-3f863cbfca13\") " pod="opendatahub/test-curl-7023-ea6cff" Oct 31 19:34:40.587933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:40.587408 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7023-ea6cff" Oct 31 19:34:40.892278 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:40.892238 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de1f4432-5xcqs" event={"ID":"bc81cb01-6336-47e0-8d44-466ccaa83704","Type":"ContainerStarted","Data":"161d9b6e897960d783778eeda7411ac7003d3c50a3d9d9744881c72234ba4806"} Oct 31 19:34:40.892698 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:40.892307 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de1f4432-5xcqs" event={"ID":"bc81cb01-6336-47e0-8d44-466ccaa83704","Type":"ContainerStarted","Data":"9da1742eec16869516f0d309b6e5ef6b6b06bcc0320d89ed9282013fe768cc6f"} Oct 31 19:34:40.925062 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:40.925001 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de1f4432-5xcqs" podStartSLOduration=1.924984168 podStartE2EDuration="1.924984168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:40.912120443 +0000 UTC m=+915.274940814" watchObservedRunningTime="2026-10-31 19:34:40.924984168 +0000 UTC m=+915.287804542" Oct 31 19:34:40.974154 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:34:40.974126 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2536d01a_00ee_4480_8462_3f863cbfca13.slice/crio-9aa087994906eec2a1bc8440abb0d70d143cfb91b476388e7f5ba085d7fe9f4d WatchSource:0}: Error finding container 9aa087994906eec2a1bc8440abb0d70d143cfb91b476388e7f5ba085d7fe9f4d: Status 404 returned error can't find the container with id 9aa087994906eec2a1bc8440abb0d70d143cfb91b476388e7f5ba085d7fe9f4d Oct 31 19:34:40.978614 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:40.977003 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7023-ea6cff"] Oct 31 19:34:41.904893 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:41.904854 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7023-ea6cff" event={"ID":"2536d01a-00ee-4480-8462-3f863cbfca13","Type":"ContainerStarted","Data":"965a61c8065c5452e60efd13c723e100307931abeff072a69cbedfb638f423ff"} Oct 31 19:34:41.905374 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:41.904921 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7023-ea6cff" event={"ID":"2536d01a-00ee-4480-8462-3f863cbfca13","Type":"ContainerStarted","Data":"9aa087994906eec2a1bc8440abb0d70d143cfb91b476388e7f5ba085d7fe9f4d"} Oct 31 19:34:41.906895 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:41.906863 4910 generic.go:356] "Generic (PLEG): container finished" podID="bc81cb01-6336-47e0-8d44-466ccaa83704" containerID="161d9b6e897960d783778eeda7411ac7003d3c50a3d9d9744881c72234ba4806" exitCode=0 Oct 31 19:34:41.907058 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:41.906952 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de1f4432-5xcqs" event={"ID":"bc81cb01-6336-47e0-8d44-466ccaa83704","Type":"ContainerDied","Data":"161d9b6e897960d783778eeda7411ac7003d3c50a3d9d9744881c72234ba4806"} Oct 31 19:34:41.925438 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:41.925381 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7023-ea6cff" podStartSLOduration=1.925368314 podStartE2EDuration="1.925368314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:41.924120343 +0000 UTC m=+916.286940739" watchObservedRunningTime="2026-10-31 19:34:41.925368314 +0000 UTC m=+916.288188686" Oct 31 19:34:42.687829 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:42.687778 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7023-ea6cff"] Oct 31 19:34:42.696697 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:42.696663 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7023-ea6cff"] Oct 31 19:34:42.787570 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:34:42.787525 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:34:43.113168 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.113138 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de1f4432-5xcqs" Oct 31 19:34:43.179387 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.179356 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc81cb01-6336-47e0-8d44-466ccaa83704-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc81cb01-6336-47e0-8d44-466ccaa83704-maas-api-xks-ca\") pod \"bc81cb01-6336-47e0-8d44-466ccaa83704\" (UID: \"bc81cb01-6336-47e0-8d44-466ccaa83704\") " Oct 31 19:34:43.179553 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.179475 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc81cb01-6336-47e0-8d44-466ccaa83704-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc81cb01-6336-47e0-8d44-466ccaa83704-maas-api-openshift-ca\") pod \"bc81cb01-6336-47e0-8d44-466ccaa83704\" (UID: \"bc81cb01-6336-47e0-8d44-466ccaa83704\") " Oct 31 19:34:43.179646 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.179559 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc81cb01-6336-47e0-8d44-466ccaa83704-kube-api-access-br75p\" (UniqueName: \"kubernetes.io/projected/bc81cb01-6336-47e0-8d44-466ccaa83704-kube-api-access-br75p\") pod \"bc81cb01-6336-47e0-8d44-466ccaa83704\" (UID: \"bc81cb01-6336-47e0-8d44-466ccaa83704\") " Oct 31 19:34:43.179903 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.179874 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc81cb01-6336-47e0-8d44-466ccaa83704-maas-api-openshift-ca" pod "bc81cb01-6336-47e0-8d44-466ccaa83704" (UID: "bc81cb01-6336-47e0-8d44-466ccaa83704"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:34:43.180430 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.180406 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc81cb01-6336-47e0-8d44-466ccaa83704-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:43.181745 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.181725 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc81cb01-6336-47e0-8d44-466ccaa83704-kube-api-access-br75p" pod "bc81cb01-6336-47e0-8d44-466ccaa83704" (UID: "bc81cb01-6336-47e0-8d44-466ccaa83704"). InnerVolumeSpecName "kube-api-access-br75p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:34:43.182070 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.182054 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc81cb01-6336-47e0-8d44-466ccaa83704-maas-api-xks-ca" pod "bc81cb01-6336-47e0-8d44-466ccaa83704" (UID: "bc81cb01-6336-47e0-8d44-466ccaa83704"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:34:43.282494 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.282416 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br75p\" (UniqueName: \"kubernetes.io/projected/bc81cb01-6336-47e0-8d44-466ccaa83704-kube-api-access-br75p\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:43.282494 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.282447 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc81cb01-6336-47e0-8d44-466ccaa83704-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:43.647722 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.647684 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30a66197-cn86c"] Oct 31 19:34:43.648461 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.648405 4910 cpu_manager.go:414] "containerMap: removing container" podUID="2536d01a-00ee-4480-8462-3f863cbfca13" containerName="test-curl-7023-ea6cff" Oct 31 19:34:43.648461 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.648424 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2536d01a-00ee-4480-8462-3f863cbfca13" containerName="test-curl-7023-ea6cff" Oct 31 19:34:43.648663 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.648491 4910 cpu_manager.go:414] "containerMap: removing container" podUID="bc81cb01-6336-47e0-8d44-466ccaa83704" containerName="revoke-keys" Oct 31 19:34:43.648663 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.648500 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc81cb01-6336-47e0-8d44-466ccaa83704" containerName="revoke-keys" Oct 31 19:34:43.648765 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.648692 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc81cb01-6336-47e0-8d44-466ccaa83704" containerName="revoke-keys" Oct 31 19:34:43.648765 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.648712 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="2536d01a-00ee-4480-8462-3f863cbfca13" containerName="test-curl-7023-ea6cff" Oct 31 19:34:43.652715 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.652696 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a66197-cn86c" Oct 31 19:34:43.668302 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.668277 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30a66197-cn86c"] Oct 31 19:34:43.796303 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.796264 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxhhs\" (UniqueName: \"kubernetes.io/projected/121ea68a-20ce-4362-b833-778115f1a9d8-kube-api-access-gxhhs\") pod \"maas-api-revoke-sub-30a66197-cn86c\" (UID: \"121ea68a-20ce-4362-b833-778115f1a9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a66197-cn86c" Oct 31 19:34:43.796565 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.796453 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121ea68a-20ce-4362-b833-778115f1a9d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30a66197-cn86c\" (UID: \"121ea68a-20ce-4362-b833-778115f1a9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a66197-cn86c" Oct 31 19:34:43.796842 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.796808 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121ea68a-20ce-4362-b833-778115f1a9d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30a66197-cn86c\" (UID: \"121ea68a-20ce-4362-b833-778115f1a9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a66197-cn86c" Oct 31 19:34:43.901002 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.900520 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121ea68a-20ce-4362-b833-778115f1a9d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30a66197-cn86c\" (UID: \"121ea68a-20ce-4362-b833-778115f1a9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a66197-cn86c" Oct 31 19:34:43.901002 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.900614 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxhhs\" (UniqueName: \"kubernetes.io/projected/121ea68a-20ce-4362-b833-778115f1a9d8-kube-api-access-gxhhs\") pod \"maas-api-revoke-sub-30a66197-cn86c\" (UID: \"121ea68a-20ce-4362-b833-778115f1a9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a66197-cn86c" Oct 31 19:34:43.901002 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.900749 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121ea68a-20ce-4362-b833-778115f1a9d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30a66197-cn86c\" (UID: \"121ea68a-20ce-4362-b833-778115f1a9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a66197-cn86c" Oct 31 19:34:43.924662 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.914196 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121ea68a-20ce-4362-b833-778115f1a9d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30a66197-cn86c\" (UID: \"121ea68a-20ce-4362-b833-778115f1a9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a66197-cn86c" Oct 31 19:34:43.970155 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.970108 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7023-ea6cff" podUID="2536d01a-00ee-4480-8462-3f863cbfca13" containerName="test-curl-7023-ea6cff" containerID="cri-o://965a61c8065c5452e60efd13c723e100307931abeff072a69cbedfb638f423ff" gracePeriod=2 Oct 31 19:34:43.970501 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.970477 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de1f4432-5xcqs" Oct 31 19:34:43.971390 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.971362 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de1f4432-5xcqs" event={"ID":"bc81cb01-6336-47e0-8d44-466ccaa83704","Type":"ContainerDied","Data":"9da1742eec16869516f0d309b6e5ef6b6b06bcc0320d89ed9282013fe768cc6f"} Oct 31 19:34:43.971508 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.971415 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9da1742eec16869516f0d309b6e5ef6b6b06bcc0320d89ed9282013fe768cc6f" Oct 31 19:34:43.972078 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.972023 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121ea68a-20ce-4362-b833-778115f1a9d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30a66197-cn86c\" (UID: \"121ea68a-20ce-4362-b833-778115f1a9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a66197-cn86c" Oct 31 19:34:43.974119 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.974089 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxhhs\" (UniqueName: \"kubernetes.io/projected/121ea68a-20ce-4362-b833-778115f1a9d8-kube-api-access-gxhhs\") pod \"maas-api-revoke-sub-30a66197-cn86c\" (UID: \"121ea68a-20ce-4362-b833-778115f1a9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a66197-cn86c" Oct 31 19:34:43.980225 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:43.980193 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a66197-cn86c" Oct 31 19:34:45.376934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:45.353875 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30a66197-cn86c"] Oct 31 19:34:45.964950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:45.944987 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:34:46.215459 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:46.209856 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a66197-cn86c" event={"ID":"121ea68a-20ce-4362-b833-778115f1a9d8","Type":"ContainerStarted","Data":"ba2c6a81d6ec6e2ab1650a5f221cf630281a21d92ec674510a95811b541a9ceb"} Oct 31 19:34:46.257960 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:46.253876 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:34:46.278972 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:46.278054 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:46.329421 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:46.322250 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 19:34:46.503194 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:46.503086 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:46.506367 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:46.504080 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:34:46.527933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:46.507233 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:46.527933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:46.507508 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:34:46.527933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:46.524361 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:46.585587 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:46.585540 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 19:34:46.968581 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:46.968534 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:34:46.971150 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:46.971124 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:34:47.078979 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:47.078001 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:47.184100 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:47.110746 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:47.184100 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:47.155016 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:47.184100 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:47.179016 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:47.210055 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:47.210003 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:47.262018 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:47.260041 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:47.262018 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:47.260482 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:34:47.278312 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:47.278271 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:47.304419 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:47.304383 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:34:47.395306 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:47.372153 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 19:34:47.483672 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:47.483638 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:34:47.487054 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:47.487025 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:34:47.512254 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:47.512165 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:34:47.513135 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:47.513110 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:34:47.533277 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:47.514427 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:47.534770 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:47.534727 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:34:47.558759 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:47.543994 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:34:47.730935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:47.544256 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:34:47.730935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:47.725565 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:34:47.730935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:47.725635 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:34:48.031155 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:48.031104 4910 generic.go:356] "Generic (PLEG): container finished" podID="2536d01a-00ee-4480-8462-3f863cbfca13" containerID="965a61c8065c5452e60efd13c723e100307931abeff072a69cbedfb638f423ff" exitCode=137 Oct 31 19:34:48.128166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:48.128117 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:48.142881 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:48.130336 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:34:48.155347 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:48.155301 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:48.157228 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:48.157200 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:34:48.263167 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:48.263069 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:34:48.313046 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:48.304572 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:34:48.449623 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:48.448902 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a66197-cn86c" event={"ID":"121ea68a-20ce-4362-b833-778115f1a9d8","Type":"ContainerStarted","Data":"7c3495796b13f4ef5dc6c9200ae3f011863eef208c684cd3cebae59c114d4882"} Oct 31 19:34:48.581951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:48.458068 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:34:48.644958 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:48.507026 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 19:34:48.644958 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:48.638053 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:34:48.644958 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:48.525756 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:48.644958 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:48.526178 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:34:48.644958 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:48.526124 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:48.644958 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:48.546102 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7023-ea6cff" Oct 31 19:34:48.644958 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:48.548102 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:48.644958 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:48.567587 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:48.644958 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:48.541789 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:34:48.782939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:48.759224 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2536d01a-00ee-4480-8462-3f863cbfca13-kube-api-access-2q8kv\" (UniqueName: \"kubernetes.io/projected/2536d01a-00ee-4480-8462-3f863cbfca13-kube-api-access-2q8kv\") pod \"2536d01a-00ee-4480-8462-3f863cbfca13\" (UID: \"2536d01a-00ee-4480-8462-3f863cbfca13\") " Oct 31 19:34:48.979937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:48.963135 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2536d01a-00ee-4480-8462-3f863cbfca13-kube-api-access-2q8kv" pod "2536d01a-00ee-4480-8462-3f863cbfca13" (UID: "2536d01a-00ee-4480-8462-3f863cbfca13"). InnerVolumeSpecName "kube-api-access-2q8kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:34:48.981254 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:48.981223 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q8kv\" (UniqueName: \"kubernetes.io/projected/2536d01a-00ee-4480-8462-3f863cbfca13-kube-api-access-2q8kv\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:49.200997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:49.198202 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:49.200997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:49.199017 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:34:49.271941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:49.266299 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:49.325116 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:49.319022 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:34:49.501984 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:49.465631 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:49.699937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:49.546761 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:49.720366 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:49.655073 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 19:34:49.723439 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:49.655078 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:49.761996 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:49.655137 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:34:49.763714 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:49.763687 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:34:49.798277 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:49.656213 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:49.798277 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:49.656239 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:34:49.798277 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:49.656258 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:49.798277 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:49.655177 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:34:50.230063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:50.230006 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:34:50.270055 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:50.270011 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:50.309938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:50.282875 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:50.339444 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:50.339389 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:50.522180 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:50.522071 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:50.544194 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:50.544159 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7023-ea6cff" Oct 31 19:34:50.586557 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:50.586515 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:34:50.586802 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:50.586772 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" podUID="16761f60-662e-4eb1-a247-b00378149cc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:50.728962 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:50.716059 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 19:34:50.728962 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:50.716446 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:34:50.728962 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:50.717201 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:34:50.746948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:50.742050 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:50.829085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:50.829040 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:51.048971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:50.840035 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:34:51.048971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:50.840076 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:34:51.048971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:50.840102 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:34:51.048971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:50.840165 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:51.048971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:50.882536 4910 generic.go:356] "Generic (PLEG): container finished" podID="121ea68a-20ce-4362-b833-778115f1a9d8" containerID="7c3495796b13f4ef5dc6c9200ae3f011863eef208c684cd3cebae59c114d4882" exitCode=0 Oct 31 19:34:51.081506 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:50.969720 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2536d01a-00ee-4480-8462-3f863cbfca13" path="/var/lib/kubelet/pods/2536d01a-00ee-4480-8462-3f863cbfca13/volumes" Oct 31 19:34:51.081506 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:34:51.074930 4910 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.311s" Oct 31 19:34:51.081506 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:51.075018 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:34:51.081506 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:51.075065 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:34:51.081506 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:51.075084 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:34:51.081506 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:51.075117 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:34:51.081506 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:51.075199 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:34:51.081506 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:51.075223 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a66197-cn86c" event={"ID":"121ea68a-20ce-4362-b833-778115f1a9d8","Type":"ContainerDied","Data":"7c3495796b13f4ef5dc6c9200ae3f011863eef208c684cd3cebae59c114d4882"} Oct 31 19:34:51.081506 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:51.075264 4910 scope.go:122] "RemoveContainer" containerID="965a61c8065c5452e60efd13c723e100307931abeff072a69cbedfb638f423ff" Oct 31 19:34:51.289609 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:51.281317 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:51.317946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:51.312442 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:34:51.357989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:51.349281 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:34:51.357989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:51.349738 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:51.736846 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:51.736794 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:51.753484 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:51.740338 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:34:51.776052 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:51.776010 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 19:34:51.870059 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:51.869962 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:34:52.060796 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.060745 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:34:52.061423 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.061392 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded" Oct 31 19:34:52.061601 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.061585 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:34:52.061828 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.061812 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:34:52.062080 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.062062 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:34:52.070628 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.065214 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": context deadline exceeded" Oct 31 19:34:52.192568 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.189764 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:34:52.192568 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.189888 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:34:52.192568 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.189954 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:34:52.198941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.193116 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:34:52.213232 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.213202 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:34:52.256626 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.249211 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:34:52.256626 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.255292 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:34:52.261312 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.261274 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:34:52.267311 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.267274 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:34:52.317849 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.301730 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:34:52.321469 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.321443 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:34:52.368779 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.368652 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:34:52.368779 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.368746 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:34:52.395961 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.395238 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c562c27-g57ct"] Oct 31 19:34:52.402535 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.402506 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c562c27-g57ct"] Oct 31 19:34:52.402685 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.402605 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c562c27-g57ct" Oct 31 19:34:52.449669 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.449589 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3899824-6ld2k"] Oct 31 19:34:52.458384 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.458357 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3899824-6ld2k" Oct 31 19:34:52.459012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.458950 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3899824-6ld2k"] Oct 31 19:34:52.530169 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.530136 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f259189-9a2f-4b33-9fd6-b9595ec217ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3899824-6ld2k\" (UID: \"2f259189-9a2f-4b33-9fd6-b9595ec217ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3899824-6ld2k" Oct 31 19:34:52.530354 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.530229 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028abb1d-a374-4439-99d6-b0e13ed729dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c562c27-g57ct\" (UID: \"028abb1d-a374-4439-99d6-b0e13ed729dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c562c27-g57ct" Oct 31 19:34:52.530436 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.530418 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f259189-9a2f-4b33-9fd6-b9595ec217ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3899824-6ld2k\" (UID: \"2f259189-9a2f-4b33-9fd6-b9595ec217ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3899824-6ld2k" Oct 31 19:34:52.530515 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.530462 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnbwn\" (UniqueName: \"kubernetes.io/projected/028abb1d-a374-4439-99d6-b0e13ed729dc-kube-api-access-dnbwn\") pod \"maas-api-revoke-sub-9c562c27-g57ct\" (UID: \"028abb1d-a374-4439-99d6-b0e13ed729dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c562c27-g57ct" Oct 31 19:34:52.530573 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.530558 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kx9t\" (UniqueName: \"kubernetes.io/projected/2f259189-9a2f-4b33-9fd6-b9595ec217ef-kube-api-access-2kx9t\") pod \"maas-api-revoke-sub-a3899824-6ld2k\" (UID: \"2f259189-9a2f-4b33-9fd6-b9595ec217ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3899824-6ld2k" Oct 31 19:34:52.530658 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.530644 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028abb1d-a374-4439-99d6-b0e13ed729dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c562c27-g57ct\" (UID: \"028abb1d-a374-4439-99d6-b0e13ed729dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c562c27-g57ct" Oct 31 19:34:52.589527 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.589439 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a66197-cn86c" Oct 31 19:34:52.631860 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.631818 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/121ea68a-20ce-4362-b833-778115f1a9d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121ea68a-20ce-4362-b833-778115f1a9d8-maas-api-openshift-ca\") pod \"121ea68a-20ce-4362-b833-778115f1a9d8\" (UID: \"121ea68a-20ce-4362-b833-778115f1a9d8\") " Oct 31 19:34:52.632473 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.632290 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/121ea68a-20ce-4362-b833-778115f1a9d8-maas-api-openshift-ca" pod "121ea68a-20ce-4362-b833-778115f1a9d8" (UID: "121ea68a-20ce-4362-b833-778115f1a9d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:34:52.632738 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.632683 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/121ea68a-20ce-4362-b833-778115f1a9d8-kube-api-access-gxhhs\" (UniqueName: \"kubernetes.io/projected/121ea68a-20ce-4362-b833-778115f1a9d8-kube-api-access-gxhhs\") pod \"121ea68a-20ce-4362-b833-778115f1a9d8\" (UID: \"121ea68a-20ce-4362-b833-778115f1a9d8\") " Oct 31 19:34:52.632867 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.632851 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/121ea68a-20ce-4362-b833-778115f1a9d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121ea68a-20ce-4362-b833-778115f1a9d8-maas-api-xks-ca\") pod \"121ea68a-20ce-4362-b833-778115f1a9d8\" (UID: \"121ea68a-20ce-4362-b833-778115f1a9d8\") " Oct 31 19:34:52.634061 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.633534 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028abb1d-a374-4439-99d6-b0e13ed729dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c562c27-g57ct\" (UID: \"028abb1d-a374-4439-99d6-b0e13ed729dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c562c27-g57ct" Oct 31 19:34:52.634320 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.634297 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f259189-9a2f-4b33-9fd6-b9595ec217ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3899824-6ld2k\" (UID: \"2f259189-9a2f-4b33-9fd6-b9595ec217ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3899824-6ld2k" Oct 31 19:34:52.634482 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.634468 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028abb1d-a374-4439-99d6-b0e13ed729dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c562c27-g57ct\" (UID: \"028abb1d-a374-4439-99d6-b0e13ed729dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c562c27-g57ct" Oct 31 19:34:52.634764 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.634746 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f259189-9a2f-4b33-9fd6-b9595ec217ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3899824-6ld2k\" (UID: \"2f259189-9a2f-4b33-9fd6-b9595ec217ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3899824-6ld2k" Oct 31 19:34:52.634879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.634866 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnbwn\" (UniqueName: \"kubernetes.io/projected/028abb1d-a374-4439-99d6-b0e13ed729dc-kube-api-access-dnbwn\") pod \"maas-api-revoke-sub-9c562c27-g57ct\" (UID: \"028abb1d-a374-4439-99d6-b0e13ed729dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c562c27-g57ct" Oct 31 19:34:52.635110 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.635095 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kx9t\" (UniqueName: \"kubernetes.io/projected/2f259189-9a2f-4b33-9fd6-b9595ec217ef-kube-api-access-2kx9t\") pod \"maas-api-revoke-sub-a3899824-6ld2k\" (UID: \"2f259189-9a2f-4b33-9fd6-b9595ec217ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3899824-6ld2k" Oct 31 19:34:52.635283 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.635270 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121ea68a-20ce-4362-b833-778115f1a9d8-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:52.637893 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.636426 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/121ea68a-20ce-4362-b833-778115f1a9d8-kube-api-access-gxhhs" pod "121ea68a-20ce-4362-b833-778115f1a9d8" (UID: "121ea68a-20ce-4362-b833-778115f1a9d8"). InnerVolumeSpecName "kube-api-access-gxhhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:34:52.637893 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.637178 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028abb1d-a374-4439-99d6-b0e13ed729dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c562c27-g57ct\" (UID: \"028abb1d-a374-4439-99d6-b0e13ed729dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c562c27-g57ct" Oct 31 19:34:52.643902 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.640224 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/121ea68a-20ce-4362-b833-778115f1a9d8-maas-api-xks-ca" pod "121ea68a-20ce-4362-b833-778115f1a9d8" (UID: "121ea68a-20ce-4362-b833-778115f1a9d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:34:52.643902 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.641500 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028abb1d-a374-4439-99d6-b0e13ed729dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c562c27-g57ct\" (UID: \"028abb1d-a374-4439-99d6-b0e13ed729dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c562c27-g57ct" Oct 31 19:34:52.643902 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.642194 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f259189-9a2f-4b33-9fd6-b9595ec217ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3899824-6ld2k\" (UID: \"2f259189-9a2f-4b33-9fd6-b9595ec217ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3899824-6ld2k" Oct 31 19:34:52.646975 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.645569 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f259189-9a2f-4b33-9fd6-b9595ec217ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3899824-6ld2k\" (UID: \"2f259189-9a2f-4b33-9fd6-b9595ec217ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3899824-6ld2k" Oct 31 19:34:52.663309 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.663033 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kx9t\" (UniqueName: \"kubernetes.io/projected/2f259189-9a2f-4b33-9fd6-b9595ec217ef-kube-api-access-2kx9t\") pod \"maas-api-revoke-sub-a3899824-6ld2k\" (UID: \"2f259189-9a2f-4b33-9fd6-b9595ec217ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3899824-6ld2k" Oct 31 19:34:52.669654 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.668743 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnbwn\" (UniqueName: \"kubernetes.io/projected/028abb1d-a374-4439-99d6-b0e13ed729dc-kube-api-access-dnbwn\") pod \"maas-api-revoke-sub-9c562c27-g57ct\" (UID: \"028abb1d-a374-4439-99d6-b0e13ed729dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c562c27-g57ct" Oct 31 19:34:52.722719 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.722627 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c562c27-g57ct" Oct 31 19:34:52.738348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.737538 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxhhs\" (UniqueName: \"kubernetes.io/projected/121ea68a-20ce-4362-b833-778115f1a9d8-kube-api-access-gxhhs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:52.738348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.737572 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121ea68a-20ce-4362-b833-778115f1a9d8-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:52.781209 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:52.780987 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3899824-6ld2k" Oct 31 19:34:53.057967 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:34:53.057932 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028abb1d_a374_4439_99d6_b0e13ed729dc.slice/crio-340a0302c707ab902df7ff82b7544e11f469ecf00fd89921cd3fd8d888daa4a0 WatchSource:0}: Error finding container 340a0302c707ab902df7ff82b7544e11f469ecf00fd89921cd3fd8d888daa4a0: Status 404 returned error can't find the container with id 340a0302c707ab902df7ff82b7544e11f469ecf00fd89921cd3fd8d888daa4a0 Oct 31 19:34:53.065022 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:53.064947 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c562c27-g57ct"] Oct 31 19:34:53.140244 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:34:53.139169 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f259189_9a2f_4b33_9fd6_b9595ec217ef.slice/crio-073e43c388ec0f90cf20f75dec95a7f27e6694401840dff2f66c5770b167d2b0 WatchSource:0}: Error finding container 073e43c388ec0f90cf20f75dec95a7f27e6694401840dff2f66c5770b167d2b0: Status 404 returned error can't find the container with id 073e43c388ec0f90cf20f75dec95a7f27e6694401840dff2f66c5770b167d2b0 Oct 31 19:34:53.173139 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:53.172485 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3899824-6ld2k"] Oct 31 19:34:53.380346 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:53.380306 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3899824-6ld2k" event={"ID":"2f259189-9a2f-4b33-9fd6-b9595ec217ef","Type":"ContainerStarted","Data":"073e43c388ec0f90cf20f75dec95a7f27e6694401840dff2f66c5770b167d2b0"} Oct 31 19:34:53.381692 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:53.381663 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c562c27-g57ct" event={"ID":"028abb1d-a374-4439-99d6-b0e13ed729dc","Type":"ContainerStarted","Data":"340a0302c707ab902df7ff82b7544e11f469ecf00fd89921cd3fd8d888daa4a0"} Oct 31 19:34:53.383208 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:53.383181 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a66197-cn86c" event={"ID":"121ea68a-20ce-4362-b833-778115f1a9d8","Type":"ContainerDied","Data":"ba2c6a81d6ec6e2ab1650a5f221cf630281a21d92ec674510a95811b541a9ceb"} Oct 31 19:34:53.383289 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:53.383224 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30a66197-cn86c" Oct 31 19:34:53.383289 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:53.383232 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba2c6a81d6ec6e2ab1650a5f221cf630281a21d92ec674510a95811b541a9ceb" Oct 31 19:34:54.394195 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:54.394160 4910 generic.go:356] "Generic (PLEG): container finished" podID="2f259189-9a2f-4b33-9fd6-b9595ec217ef" containerID="8fd6ef98fd815b05f578dcd71cd6f3149b518c87f582ecabc658901d64b35528" exitCode=0 Oct 31 19:34:54.394597 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:54.394246 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3899824-6ld2k" event={"ID":"2f259189-9a2f-4b33-9fd6-b9595ec217ef","Type":"ContainerDied","Data":"8fd6ef98fd815b05f578dcd71cd6f3149b518c87f582ecabc658901d64b35528"} Oct 31 19:34:54.396008 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:54.395985 4910 generic.go:356] "Generic (PLEG): container finished" podID="028abb1d-a374-4439-99d6-b0e13ed729dc" containerID="d40a2731d2e1ceb6bddcad5fbd1d49993fdc9c2aedf8bf5280fd43f6b67ed8ac" exitCode=0 Oct 31 19:34:54.396142 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:54.396080 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c562c27-g57ct" event={"ID":"028abb1d-a374-4439-99d6-b0e13ed729dc","Type":"ContainerDied","Data":"d40a2731d2e1ceb6bddcad5fbd1d49993fdc9c2aedf8bf5280fd43f6b67ed8ac"} Oct 31 19:34:55.681609 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:55.681536 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3899824-6ld2k" Oct 31 19:34:55.775821 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:55.775782 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c562c27-g57ct" Oct 31 19:34:55.815345 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:55.815306 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f259189-9a2f-4b33-9fd6-b9595ec217ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f259189-9a2f-4b33-9fd6-b9595ec217ef-maas-api-xks-ca\") pod \"2f259189-9a2f-4b33-9fd6-b9595ec217ef\" (UID: \"2f259189-9a2f-4b33-9fd6-b9595ec217ef\") " Oct 31 19:34:55.815519 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:55.815418 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f259189-9a2f-4b33-9fd6-b9595ec217ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f259189-9a2f-4b33-9fd6-b9595ec217ef-maas-api-openshift-ca\") pod \"2f259189-9a2f-4b33-9fd6-b9595ec217ef\" (UID: \"2f259189-9a2f-4b33-9fd6-b9595ec217ef\") " Oct 31 19:34:55.815519 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:55.815450 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f259189-9a2f-4b33-9fd6-b9595ec217ef-kube-api-access-2kx9t\" (UniqueName: \"kubernetes.io/projected/2f259189-9a2f-4b33-9fd6-b9595ec217ef-kube-api-access-2kx9t\") pod \"2f259189-9a2f-4b33-9fd6-b9595ec217ef\" (UID: \"2f259189-9a2f-4b33-9fd6-b9595ec217ef\") " Oct 31 19:34:55.816320 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:55.816291 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f259189-9a2f-4b33-9fd6-b9595ec217ef-maas-api-openshift-ca" pod "2f259189-9a2f-4b33-9fd6-b9595ec217ef" (UID: "2f259189-9a2f-4b33-9fd6-b9595ec217ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:34:55.820438 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:55.820387 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f259189-9a2f-4b33-9fd6-b9595ec217ef-kube-api-access-2kx9t" pod "2f259189-9a2f-4b33-9fd6-b9595ec217ef" (UID: "2f259189-9a2f-4b33-9fd6-b9595ec217ef"). InnerVolumeSpecName "kube-api-access-2kx9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:34:55.823086 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:55.823059 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f259189-9a2f-4b33-9fd6-b9595ec217ef-maas-api-xks-ca" pod "2f259189-9a2f-4b33-9fd6-b9595ec217ef" (UID: "2f259189-9a2f-4b33-9fd6-b9595ec217ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:34:55.917239 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:55.917201 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/028abb1d-a374-4439-99d6-b0e13ed729dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028abb1d-a374-4439-99d6-b0e13ed729dc-maas-api-openshift-ca\") pod \"028abb1d-a374-4439-99d6-b0e13ed729dc\" (UID: \"028abb1d-a374-4439-99d6-b0e13ed729dc\") " Oct 31 19:34:55.917425 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:55.917326 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/028abb1d-a374-4439-99d6-b0e13ed729dc-kube-api-access-dnbwn\" (UniqueName: \"kubernetes.io/projected/028abb1d-a374-4439-99d6-b0e13ed729dc-kube-api-access-dnbwn\") pod \"028abb1d-a374-4439-99d6-b0e13ed729dc\" (UID: \"028abb1d-a374-4439-99d6-b0e13ed729dc\") " Oct 31 19:34:55.917487 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:55.917464 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/028abb1d-a374-4439-99d6-b0e13ed729dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028abb1d-a374-4439-99d6-b0e13ed729dc-maas-api-xks-ca\") pod \"028abb1d-a374-4439-99d6-b0e13ed729dc\" (UID: \"028abb1d-a374-4439-99d6-b0e13ed729dc\") " Oct 31 19:34:55.917553 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:55.917528 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/028abb1d-a374-4439-99d6-b0e13ed729dc-maas-api-openshift-ca" pod "028abb1d-a374-4439-99d6-b0e13ed729dc" (UID: "028abb1d-a374-4439-99d6-b0e13ed729dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:34:55.918350 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:55.918319 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f259189-9a2f-4b33-9fd6-b9595ec217ef-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:55.918350 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:55.918348 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/028abb1d-a374-4439-99d6-b0e13ed729dc-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:55.918551 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:55.918363 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f259189-9a2f-4b33-9fd6-b9595ec217ef-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:55.918551 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:55.918376 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kx9t\" (UniqueName: \"kubernetes.io/projected/2f259189-9a2f-4b33-9fd6-b9595ec217ef-kube-api-access-2kx9t\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:55.919497 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:55.919474 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/028abb1d-a374-4439-99d6-b0e13ed729dc-maas-api-xks-ca" pod "028abb1d-a374-4439-99d6-b0e13ed729dc" (UID: "028abb1d-a374-4439-99d6-b0e13ed729dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:34:55.919826 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:55.919802 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/028abb1d-a374-4439-99d6-b0e13ed729dc-kube-api-access-dnbwn" pod "028abb1d-a374-4439-99d6-b0e13ed729dc" (UID: "028abb1d-a374-4439-99d6-b0e13ed729dc"). InnerVolumeSpecName "kube-api-access-dnbwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:34:56.020299 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.020211 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/028abb1d-a374-4439-99d6-b0e13ed729dc-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:56.020299 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.020249 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnbwn\" (UniqueName: \"kubernetes.io/projected/028abb1d-a374-4439-99d6-b0e13ed729dc-kube-api-access-dnbwn\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:56.492944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.488032 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3899824-6ld2k" event={"ID":"2f259189-9a2f-4b33-9fd6-b9595ec217ef","Type":"ContainerDied","Data":"073e43c388ec0f90cf20f75dec95a7f27e6694401840dff2f66c5770b167d2b0"} Oct 31 19:34:56.492944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.488100 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="073e43c388ec0f90cf20f75dec95a7f27e6694401840dff2f66c5770b167d2b0" Oct 31 19:34:56.492944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.488188 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3899824-6ld2k" Oct 31 19:34:56.504946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.498336 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c562c27-g57ct" event={"ID":"028abb1d-a374-4439-99d6-b0e13ed729dc","Type":"ContainerDied","Data":"340a0302c707ab902df7ff82b7544e11f469ecf00fd89921cd3fd8d888daa4a0"} Oct 31 19:34:56.505186 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.505165 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="340a0302c707ab902df7ff82b7544e11f469ecf00fd89921cd3fd8d888daa4a0" Oct 31 19:34:56.505380 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.505366 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c562c27-g57ct" Oct 31 19:34:56.683770 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.683736 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp"] Oct 31 19:34:56.684626 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.684603 4910 cpu_manager.go:414] "containerMap: removing container" podUID="121ea68a-20ce-4362-b833-778115f1a9d8" containerName="revoke-keys" Oct 31 19:34:56.684626 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.684627 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="121ea68a-20ce-4362-b833-778115f1a9d8" containerName="revoke-keys" Oct 31 19:34:56.684813 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.684646 4910 cpu_manager.go:414] "containerMap: removing container" podUID="2f259189-9a2f-4b33-9fd6-b9595ec217ef" containerName="revoke-keys" Oct 31 19:34:56.684813 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.684653 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f259189-9a2f-4b33-9fd6-b9595ec217ef" containerName="revoke-keys" Oct 31 19:34:56.684813 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.684669 4910 cpu_manager.go:414] "containerMap: removing container" podUID="028abb1d-a374-4439-99d6-b0e13ed729dc" containerName="revoke-keys" Oct 31 19:34:56.684813 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.684676 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="028abb1d-a374-4439-99d6-b0e13ed729dc" containerName="revoke-keys" Oct 31 19:34:56.685356 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.684943 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="121ea68a-20ce-4362-b833-778115f1a9d8" containerName="revoke-keys" Oct 31 19:34:56.685356 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.684963 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f259189-9a2f-4b33-9fd6-b9595ec217ef" containerName="revoke-keys" Oct 31 19:34:56.685356 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.684976 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="028abb1d-a374-4439-99d6-b0e13ed729dc" containerName="revoke-keys" Oct 31 19:34:56.689350 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.689327 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp" Oct 31 19:34:56.695966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.695806 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:34:56.695966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.695875 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:34:56.704641 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.704617 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp"] Oct 31 19:34:56.853750 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.853465 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/692d071f-24ea-4db2-a615-a7e4f4b85631-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp\" (UID: \"692d071f-24ea-4db2-a615-a7e4f4b85631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp" Oct 31 19:34:56.853750 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.853589 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/692d071f-24ea-4db2-a615-a7e4f4b85631-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp\" (UID: \"692d071f-24ea-4db2-a615-a7e4f4b85631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp" Oct 31 19:34:56.854073 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.854003 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qwfd\" (UniqueName: \"kubernetes.io/projected/692d071f-24ea-4db2-a615-a7e4f4b85631-kube-api-access-7qwfd\") pod \"maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp\" (UID: \"692d071f-24ea-4db2-a615-a7e4f4b85631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp" Oct 31 19:34:56.956047 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.956003 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/692d071f-24ea-4db2-a615-a7e4f4b85631-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp\" (UID: \"692d071f-24ea-4db2-a615-a7e4f4b85631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp" Oct 31 19:34:56.956375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.956358 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qwfd\" (UniqueName: \"kubernetes.io/projected/692d071f-24ea-4db2-a615-a7e4f4b85631-kube-api-access-7qwfd\") pod \"maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp\" (UID: \"692d071f-24ea-4db2-a615-a7e4f4b85631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp" Oct 31 19:34:56.956452 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.956404 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/692d071f-24ea-4db2-a615-a7e4f4b85631-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp\" (UID: \"692d071f-24ea-4db2-a615-a7e4f4b85631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp" Oct 31 19:34:56.957224 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.957200 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/692d071f-24ea-4db2-a615-a7e4f4b85631-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp\" (UID: \"692d071f-24ea-4db2-a615-a7e4f4b85631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp" Oct 31 19:34:56.958605 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.958575 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/692d071f-24ea-4db2-a615-a7e4f4b85631-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp\" (UID: \"692d071f-24ea-4db2-a615-a7e4f4b85631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp" Oct 31 19:34:56.969363 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:56.969330 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qwfd\" (UniqueName: \"kubernetes.io/projected/692d071f-24ea-4db2-a615-a7e4f4b85631-kube-api-access-7qwfd\") pod \"maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp\" (UID: \"692d071f-24ea-4db2-a615-a7e4f4b85631\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp" Oct 31 19:34:57.006804 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:57.006779 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp" Oct 31 19:34:57.304512 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:34:57.304482 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692d071f_24ea_4db2_a615_a7e4f4b85631.slice/crio-95950ecb38aec58cf661088086c75683714e3f3048e3074ea94e39f80bc7a06f WatchSource:0}: Error finding container 95950ecb38aec58cf661088086c75683714e3f3048e3074ea94e39f80bc7a06f: Status 404 returned error can't find the container with id 95950ecb38aec58cf661088086c75683714e3f3048e3074ea94e39f80bc7a06f Oct 31 19:34:57.307544 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:57.307519 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp"] Oct 31 19:34:57.509298 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:57.509266 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp" event={"ID":"692d071f-24ea-4db2-a615-a7e4f4b85631","Type":"ContainerStarted","Data":"95950ecb38aec58cf661088086c75683714e3f3048e3074ea94e39f80bc7a06f"} Oct 31 19:34:58.521676 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:58.521595 4910 generic.go:356] "Generic (PLEG): container finished" podID="692d071f-24ea-4db2-a615-a7e4f4b85631" containerID="cc71dc7e40aab32e4c594451954bed8b145a7c986df063f9def589efb096bac5" exitCode=0 Oct 31 19:34:58.521676 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:58.521633 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp" event={"ID":"692d071f-24ea-4db2-a615-a7e4f4b85631","Type":"ContainerDied","Data":"cc71dc7e40aab32e4c594451954bed8b145a7c986df063f9def589efb096bac5"} Oct 31 19:34:59.738322 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:59.738297 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp" Oct 31 19:34:59.803648 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:59.803555 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/692d071f-24ea-4db2-a615-a7e4f4b85631-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/692d071f-24ea-4db2-a615-a7e4f4b85631-maas-api-openshift-ca\") pod \"692d071f-24ea-4db2-a615-a7e4f4b85631\" (UID: \"692d071f-24ea-4db2-a615-a7e4f4b85631\") " Oct 31 19:34:59.803791 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:59.803747 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/692d071f-24ea-4db2-a615-a7e4f4b85631-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/692d071f-24ea-4db2-a615-a7e4f4b85631-maas-api-xks-ca\") pod \"692d071f-24ea-4db2-a615-a7e4f4b85631\" (UID: \"692d071f-24ea-4db2-a615-a7e4f4b85631\") " Oct 31 19:34:59.804106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:59.804019 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692d071f-24ea-4db2-a615-a7e4f4b85631-maas-api-openshift-ca" pod "692d071f-24ea-4db2-a615-a7e4f4b85631" (UID: "692d071f-24ea-4db2-a615-a7e4f4b85631"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:34:59.805038 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:59.805014 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/692d071f-24ea-4db2-a615-a7e4f4b85631-kube-api-access-7qwfd\" (UniqueName: \"kubernetes.io/projected/692d071f-24ea-4db2-a615-a7e4f4b85631-kube-api-access-7qwfd\") pod \"692d071f-24ea-4db2-a615-a7e4f4b85631\" (UID: \"692d071f-24ea-4db2-a615-a7e4f4b85631\") " Oct 31 19:34:59.806081 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:59.806041 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/692d071f-24ea-4db2-a615-a7e4f4b85631-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:59.806512 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:59.806437 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692d071f-24ea-4db2-a615-a7e4f4b85631-maas-api-xks-ca" pod "692d071f-24ea-4db2-a615-a7e4f4b85631" (UID: "692d071f-24ea-4db2-a615-a7e4f4b85631"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:34:59.807523 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:59.807496 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692d071f-24ea-4db2-a615-a7e4f4b85631-kube-api-access-7qwfd" pod "692d071f-24ea-4db2-a615-a7e4f4b85631" (UID: "692d071f-24ea-4db2-a615-a7e4f4b85631"). InnerVolumeSpecName "kube-api-access-7qwfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:34:59.907532 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:59.907495 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qwfd\" (UniqueName: \"kubernetes.io/projected/692d071f-24ea-4db2-a615-a7e4f4b85631-kube-api-access-7qwfd\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:34:59.907532 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:34:59.907526 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/692d071f-24ea-4db2-a615-a7e4f4b85631-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:00.268933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.268086 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx"] Oct 31 19:35:00.292291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.272429 4910 cpu_manager.go:414] "containerMap: removing container" podUID="692d071f-24ea-4db2-a615-a7e4f4b85631" containerName="revoke-keys" Oct 31 19:35:00.292291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.272460 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="692d071f-24ea-4db2-a615-a7e4f4b85631" containerName="revoke-keys" Oct 31 19:35:00.292291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.272941 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="692d071f-24ea-4db2-a615-a7e4f4b85631" containerName="revoke-keys" Oct 31 19:35:00.319782 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.319742 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 19:35:00.340013 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.339969 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7z5vd\"" Oct 31 19:35:00.340300 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.340279 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 19:35:00.340531 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.340513 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 19:35:00.340732 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.340717 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 19:35:00.390446 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.390394 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx"] Oct 31 19:35:00.399308 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.399270 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhgqx\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 19:35:00.399479 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.399346 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhgqx\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 19:35:00.399479 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.399409 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk9s5\" (UniqueName: \"kubernetes.io/projected/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-kube-api-access-fk9s5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhgqx\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 19:35:00.399597 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.399519 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhgqx\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 19:35:00.399666 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.399650 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhgqx\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 19:35:00.403624 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.403596 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhgqx\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 19:35:00.403773 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.403647 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhgqx\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 19:35:00.505360 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.505323 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhgqx\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 19:35:00.505510 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.505426 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhgqx\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 19:35:00.505510 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.505454 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhgqx\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 19:35:00.505578 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.505546 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhgqx\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 19:35:00.505637 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.505587 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhgqx\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 19:35:00.505689 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.505634 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk9s5\" (UniqueName: \"kubernetes.io/projected/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-kube-api-access-fk9s5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhgqx\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 19:35:00.506087 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.505739 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhgqx\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 19:35:00.506367 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.506339 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhgqx\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 19:35:00.507956 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.507929 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhgqx\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 19:35:00.508044 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.507955 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhgqx\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 19:35:00.508436 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.508418 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhgqx\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 19:35:00.508609 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.508586 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhgqx\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 19:35:00.523171 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.523083 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk9s5\" (UniqueName: \"kubernetes.io/projected/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-kube-api-access-fk9s5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhgqx\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 19:35:00.525124 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.525099 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhgqx\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 19:35:00.549158 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.549132 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp" Oct 31 19:35:00.549158 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.549140 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp" event={"ID":"692d071f-24ea-4db2-a615-a7e4f4b85631","Type":"ContainerDied","Data":"95950ecb38aec58cf661088086c75683714e3f3048e3074ea94e39f80bc7a06f"} Oct 31 19:35:00.549329 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.549196 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95950ecb38aec58cf661088086c75683714e3f3048e3074ea94e39f80bc7a06f" Oct 31 19:35:00.702283 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:00.702242 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 19:35:01.106455 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:35:01.106427 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65cd5c60_1a89_4cd0_a754_b04cd8e46c37.slice/crio-e3f9fda7ec4597d403600f5aa7b65d69f1c75bb537f93dfb0b36c7662c8dd0a9 WatchSource:0}: Error finding container e3f9fda7ec4597d403600f5aa7b65d69f1c75bb537f93dfb0b36c7662c8dd0a9: Status 404 returned error can't find the container with id e3f9fda7ec4597d403600f5aa7b65d69f1c75bb537f93dfb0b36c7662c8dd0a9 Oct 31 19:35:01.108703 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:01.108687 4910 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 19:35:01.109223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:01.109202 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx"] Oct 31 19:35:01.559354 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:01.559308 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" event={"ID":"65cd5c60-1a89-4cd0-a754-b04cd8e46c37","Type":"ContainerStarted","Data":"e3f9fda7ec4597d403600f5aa7b65d69f1c75bb537f93dfb0b36c7662c8dd0a9"} Oct 31 19:35:01.661991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:01.661951 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl"] Oct 31 19:35:01.668365 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:01.668320 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl" Oct 31 19:35:01.674110 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:01.674082 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:35:01.674110 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:01.674086 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:35:01.677141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:01.677118 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl"] Oct 31 19:35:01.726290 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:01.726247 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg829\" (UniqueName: \"kubernetes.io/projected/e2d68baa-fee1-441b-8245-cd0083ea6d3a-kube-api-access-vg829\") pod \"maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl\" (UID: \"e2d68baa-fee1-441b-8245-cd0083ea6d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl" Oct 31 19:35:01.726495 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:01.726478 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2d68baa-fee1-441b-8245-cd0083ea6d3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl\" (UID: \"e2d68baa-fee1-441b-8245-cd0083ea6d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl" Oct 31 19:35:01.726630 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:01.726611 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2d68baa-fee1-441b-8245-cd0083ea6d3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl\" (UID: \"e2d68baa-fee1-441b-8245-cd0083ea6d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl" Oct 31 19:35:01.828238 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:01.828132 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2d68baa-fee1-441b-8245-cd0083ea6d3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl\" (UID: \"e2d68baa-fee1-441b-8245-cd0083ea6d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl" Oct 31 19:35:01.828476 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:01.828260 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2d68baa-fee1-441b-8245-cd0083ea6d3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl\" (UID: \"e2d68baa-fee1-441b-8245-cd0083ea6d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl" Oct 31 19:35:01.829041 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:01.828893 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg829\" (UniqueName: \"kubernetes.io/projected/e2d68baa-fee1-441b-8245-cd0083ea6d3a-kube-api-access-vg829\") pod \"maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl\" (UID: \"e2d68baa-fee1-441b-8245-cd0083ea6d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl" Oct 31 19:35:01.830089 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:01.830061 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2d68baa-fee1-441b-8245-cd0083ea6d3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl\" (UID: \"e2d68baa-fee1-441b-8245-cd0083ea6d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl" Oct 31 19:35:01.838426 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:01.838393 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2d68baa-fee1-441b-8245-cd0083ea6d3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl\" (UID: \"e2d68baa-fee1-441b-8245-cd0083ea6d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl" Oct 31 19:35:01.843898 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:01.843874 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg829\" (UniqueName: \"kubernetes.io/projected/e2d68baa-fee1-441b-8245-cd0083ea6d3a-kube-api-access-vg829\") pod \"maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl\" (UID: \"e2d68baa-fee1-441b-8245-cd0083ea6d3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl" Oct 31 19:35:02.007015 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:02.006972 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl" Oct 31 19:35:02.294019 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:35:02.293982 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d68baa_fee1_441b_8245_cd0083ea6d3a.slice/crio-f6a34a55749f479cbb8718bb9b7dbcd96c22a434a9212b2f9ef642d80fd0d0be WatchSource:0}: Error finding container f6a34a55749f479cbb8718bb9b7dbcd96c22a434a9212b2f9ef642d80fd0d0be: Status 404 returned error can't find the container with id f6a34a55749f479cbb8718bb9b7dbcd96c22a434a9212b2f9ef642d80fd0d0be Oct 31 19:35:02.297269 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:02.297240 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl"] Oct 31 19:35:02.573410 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:02.573374 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl" event={"ID":"e2d68baa-fee1-441b-8245-cd0083ea6d3a","Type":"ContainerStarted","Data":"f6a34a55749f479cbb8718bb9b7dbcd96c22a434a9212b2f9ef642d80fd0d0be"} Oct 31 19:35:03.590790 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:03.590747 4910 generic.go:356] "Generic (PLEG): container finished" podID="e2d68baa-fee1-441b-8245-cd0083ea6d3a" containerID="938c793e482ab097f4b2404a129f15fdc2085114d23e5feed4c8864227b63a86" exitCode=0 Oct 31 19:35:03.591333 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:03.590833 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl" event={"ID":"e2d68baa-fee1-441b-8245-cd0083ea6d3a","Type":"ContainerDied","Data":"938c793e482ab097f4b2404a129f15fdc2085114d23e5feed4c8864227b63a86"} Oct 31 19:35:04.013639 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:04.013597 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e7e2625-mgctc"] Oct 31 19:35:04.022115 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:04.022089 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7e2625-mgctc" Oct 31 19:35:04.027564 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:04.027530 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e7e2625-mgctc"] Oct 31 19:35:04.093565 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:04.093517 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwwj6\" (UniqueName: \"kubernetes.io/projected/df36c2ed-5532-4247-872c-54d13d7506a3-kube-api-access-mwwj6\") pod \"maas-api-revoke-sub-5e7e2625-mgctc\" (UID: \"df36c2ed-5532-4247-872c-54d13d7506a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7e2625-mgctc" Oct 31 19:35:04.093758 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:04.093681 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df36c2ed-5532-4247-872c-54d13d7506a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e7e2625-mgctc\" (UID: \"df36c2ed-5532-4247-872c-54d13d7506a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7e2625-mgctc" Oct 31 19:35:04.093949 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:04.093891 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df36c2ed-5532-4247-872c-54d13d7506a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e7e2625-mgctc\" (UID: \"df36c2ed-5532-4247-872c-54d13d7506a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7e2625-mgctc" Oct 31 19:35:04.195291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:04.195255 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df36c2ed-5532-4247-872c-54d13d7506a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e7e2625-mgctc\" (UID: \"df36c2ed-5532-4247-872c-54d13d7506a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7e2625-mgctc" Oct 31 19:35:04.195483 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:04.195397 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwwj6\" (UniqueName: \"kubernetes.io/projected/df36c2ed-5532-4247-872c-54d13d7506a3-kube-api-access-mwwj6\") pod \"maas-api-revoke-sub-5e7e2625-mgctc\" (UID: \"df36c2ed-5532-4247-872c-54d13d7506a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7e2625-mgctc" Oct 31 19:35:04.195602 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:04.195485 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df36c2ed-5532-4247-872c-54d13d7506a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e7e2625-mgctc\" (UID: \"df36c2ed-5532-4247-872c-54d13d7506a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7e2625-mgctc" Oct 31 19:35:04.196500 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:04.196466 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df36c2ed-5532-4247-872c-54d13d7506a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e7e2625-mgctc\" (UID: \"df36c2ed-5532-4247-872c-54d13d7506a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7e2625-mgctc" Oct 31 19:35:04.198219 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:04.198195 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df36c2ed-5532-4247-872c-54d13d7506a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e7e2625-mgctc\" (UID: \"df36c2ed-5532-4247-872c-54d13d7506a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7e2625-mgctc" Oct 31 19:35:04.209517 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:04.209494 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwwj6\" (UniqueName: \"kubernetes.io/projected/df36c2ed-5532-4247-872c-54d13d7506a3-kube-api-access-mwwj6\") pod \"maas-api-revoke-sub-5e7e2625-mgctc\" (UID: \"df36c2ed-5532-4247-872c-54d13d7506a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7e2625-mgctc" Oct 31 19:35:04.342711 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:04.342673 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7e2625-mgctc" Oct 31 19:35:04.676375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:04.675503 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e7e2625-mgctc"] Oct 31 19:35:04.775048 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:04.774622 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:35:04.775048 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:04.774702 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:35:04.945710 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:04.945684 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl" Oct 31 19:35:05.038131 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.038100 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2d68baa-fee1-441b-8245-cd0083ea6d3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2d68baa-fee1-441b-8245-cd0083ea6d3a-maas-api-openshift-ca\") pod \"e2d68baa-fee1-441b-8245-cd0083ea6d3a\" (UID: \"e2d68baa-fee1-441b-8245-cd0083ea6d3a\") " Oct 31 19:35:05.038292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.038181 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2d68baa-fee1-441b-8245-cd0083ea6d3a-kube-api-access-vg829\" (UniqueName: \"kubernetes.io/projected/e2d68baa-fee1-441b-8245-cd0083ea6d3a-kube-api-access-vg829\") pod \"e2d68baa-fee1-441b-8245-cd0083ea6d3a\" (UID: \"e2d68baa-fee1-441b-8245-cd0083ea6d3a\") " Oct 31 19:35:05.038292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.038284 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2d68baa-fee1-441b-8245-cd0083ea6d3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2d68baa-fee1-441b-8245-cd0083ea6d3a-maas-api-xks-ca\") pod \"e2d68baa-fee1-441b-8245-cd0083ea6d3a\" (UID: \"e2d68baa-fee1-441b-8245-cd0083ea6d3a\") " Oct 31 19:35:05.038533 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.038505 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2d68baa-fee1-441b-8245-cd0083ea6d3a-maas-api-openshift-ca" pod "e2d68baa-fee1-441b-8245-cd0083ea6d3a" (UID: "e2d68baa-fee1-441b-8245-cd0083ea6d3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:35:05.039007 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.038975 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2d68baa-fee1-441b-8245-cd0083ea6d3a-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:05.040339 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.040317 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2d68baa-fee1-441b-8245-cd0083ea6d3a-maas-api-xks-ca" pod "e2d68baa-fee1-441b-8245-cd0083ea6d3a" (UID: "e2d68baa-fee1-441b-8245-cd0083ea6d3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:35:05.040733 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.040720 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d68baa-fee1-441b-8245-cd0083ea6d3a-kube-api-access-vg829" pod "e2d68baa-fee1-441b-8245-cd0083ea6d3a" (UID: "e2d68baa-fee1-441b-8245-cd0083ea6d3a"). InnerVolumeSpecName "kube-api-access-vg829". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:35:05.140163 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.140137 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg829\" (UniqueName: \"kubernetes.io/projected/e2d68baa-fee1-441b-8245-cd0083ea6d3a-kube-api-access-vg829\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:05.140163 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.140163 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2d68baa-fee1-441b-8245-cd0083ea6d3a-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:05.623046 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.622981 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2"] Oct 31 19:35:05.624052 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.624029 4910 cpu_manager.go:414] "containerMap: removing container" podUID="e2d68baa-fee1-441b-8245-cd0083ea6d3a" containerName="revoke-keys" Oct 31 19:35:05.624052 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.624051 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2d68baa-fee1-441b-8245-cd0083ea6d3a" containerName="revoke-keys" Oct 31 19:35:05.624316 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.624264 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2d68baa-fee1-441b-8245-cd0083ea6d3a" containerName="revoke-keys" Oct 31 19:35:05.628664 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.628640 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2" Oct 31 19:35:05.639859 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.638762 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2"] Oct 31 19:35:05.689121 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.689087 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl" event={"ID":"e2d68baa-fee1-441b-8245-cd0083ea6d3a","Type":"ContainerDied","Data":"f6a34a55749f479cbb8718bb9b7dbcd96c22a434a9212b2f9ef642d80fd0d0be"} Oct 31 19:35:05.689678 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.689148 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6a34a55749f479cbb8718bb9b7dbcd96c22a434a9212b2f9ef642d80fd0d0be" Oct 31 19:35:05.689678 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.689230 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl" Oct 31 19:35:05.693485 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.693447 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7e2625-mgctc" event={"ID":"df36c2ed-5532-4247-872c-54d13d7506a3","Type":"ContainerStarted","Data":"c9491a09b25c87e60a2dacfc94e634611f8d7975f3f2e26489b25440cf4b66d6"} Oct 31 19:35:05.693615 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.693512 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7e2625-mgctc" event={"ID":"df36c2ed-5532-4247-872c-54d13d7506a3","Type":"ContainerStarted","Data":"3c1531ff102ae5de022a439a21221c103ad80389813d0775ee18ef42fe7913d6"} Oct 31 19:35:05.697047 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.697016 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" event={"ID":"65cd5c60-1a89-4cd0-a754-b04cd8e46c37","Type":"ContainerStarted","Data":"3d3b3e752555bc488d151235c451fed4632b4cea2738177bad47cbe6f79982be"} Oct 31 19:35:05.697258 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.697240 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 19:35:05.700217 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.700194 4910 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bhgqx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:35:05.700294 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.700236 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" podUID="65cd5c60-1a89-4cd0-a754-b04cd8e46c37" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:35:05.726518 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.726458 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7e2625-mgctc" podStartSLOduration=2.726439779 podStartE2EDuration="2.726439779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:35:05.716704461 +0000 UTC m=+940.079524832" watchObservedRunningTime="2026-10-31 19:35:05.726439779 +0000 UTC m=+940.089260148" Oct 31 19:35:05.750613 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.750555 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" podStartSLOduration=2.086843461 podStartE2EDuration="5.750539689s" totalImagesPullingTime="3.663696228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:35:05.744648082 +0000 UTC m=+940.107468454" watchObservedRunningTime="2026-10-31 19:35:05.750539689 +0000 UTC m=+940.113360062" Oct 31 19:35:05.762549 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.762514 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82722265-59cd-40d4-96ad-5cbf775a9cdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2\" (UID: \"82722265-59cd-40d4-96ad-5cbf775a9cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2" Oct 31 19:35:05.762719 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.762641 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82722265-59cd-40d4-96ad-5cbf775a9cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2\" (UID: \"82722265-59cd-40d4-96ad-5cbf775a9cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2" Oct 31 19:35:05.763062 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.762883 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g26zm\" (UniqueName: \"kubernetes.io/projected/82722265-59cd-40d4-96ad-5cbf775a9cdd-kube-api-access-g26zm\") pod \"maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2\" (UID: \"82722265-59cd-40d4-96ad-5cbf775a9cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2" Oct 31 19:35:05.865463 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.865358 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82722265-59cd-40d4-96ad-5cbf775a9cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2\" (UID: \"82722265-59cd-40d4-96ad-5cbf775a9cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2" Oct 31 19:35:05.865968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.865905 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g26zm\" (UniqueName: \"kubernetes.io/projected/82722265-59cd-40d4-96ad-5cbf775a9cdd-kube-api-access-g26zm\") pod \"maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2\" (UID: \"82722265-59cd-40d4-96ad-5cbf775a9cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2" Oct 31 19:35:05.866247 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.866221 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82722265-59cd-40d4-96ad-5cbf775a9cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2\" (UID: \"82722265-59cd-40d4-96ad-5cbf775a9cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2" Oct 31 19:35:05.866415 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.866394 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82722265-59cd-40d4-96ad-5cbf775a9cdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2\" (UID: \"82722265-59cd-40d4-96ad-5cbf775a9cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2" Oct 31 19:35:05.868678 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.868653 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82722265-59cd-40d4-96ad-5cbf775a9cdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2\" (UID: \"82722265-59cd-40d4-96ad-5cbf775a9cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2" Oct 31 19:35:05.878953 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.878852 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g26zm\" (UniqueName: \"kubernetes.io/projected/82722265-59cd-40d4-96ad-5cbf775a9cdd-kube-api-access-g26zm\") pod \"maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2\" (UID: \"82722265-59cd-40d4-96ad-5cbf775a9cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2" Oct 31 19:35:05.975957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:05.975902 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2" Oct 31 19:35:06.268422 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:35:06.268387 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82722265_59cd_40d4_96ad_5cbf775a9cdd.slice/crio-ac6fee8c6e30cea624d6b14d7359f86ba14a7889c43499ebcf4de87da5cd679e WatchSource:0}: Error finding container ac6fee8c6e30cea624d6b14d7359f86ba14a7889c43499ebcf4de87da5cd679e: Status 404 returned error can't find the container with id ac6fee8c6e30cea624d6b14d7359f86ba14a7889c43499ebcf4de87da5cd679e Oct 31 19:35:06.271660 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:06.271633 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2"] Oct 31 19:35:06.709045 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:06.709001 4910 generic.go:356] "Generic (PLEG): container finished" podID="df36c2ed-5532-4247-872c-54d13d7506a3" containerID="c9491a09b25c87e60a2dacfc94e634611f8d7975f3f2e26489b25440cf4b66d6" exitCode=0 Oct 31 19:35:06.709458 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:06.709085 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7e2625-mgctc" event={"ID":"df36c2ed-5532-4247-872c-54d13d7506a3","Type":"ContainerDied","Data":"c9491a09b25c87e60a2dacfc94e634611f8d7975f3f2e26489b25440cf4b66d6"} Oct 31 19:35:06.710804 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:06.710775 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2" event={"ID":"82722265-59cd-40d4-96ad-5cbf775a9cdd","Type":"ContainerStarted","Data":"ac6fee8c6e30cea624d6b14d7359f86ba14a7889c43499ebcf4de87da5cd679e"} Oct 31 19:35:06.711925 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:06.711891 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 19:35:07.723239 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:07.723133 4910 generic.go:356] "Generic (PLEG): container finished" podID="82722265-59cd-40d4-96ad-5cbf775a9cdd" containerID="10c1225c10a8b30bdd673d74e3f70bab6afc2312e067a8f0ab93282a1e832820" exitCode=0 Oct 31 19:35:07.723239 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:07.723214 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2" event={"ID":"82722265-59cd-40d4-96ad-5cbf775a9cdd","Type":"ContainerDied","Data":"10c1225c10a8b30bdd673d74e3f70bab6afc2312e067a8f0ab93282a1e832820"} Oct 31 19:35:07.922857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:07.922819 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7e2625-mgctc" Oct 31 19:35:07.937285 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:07.937253 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df36c2ed-5532-4247-872c-54d13d7506a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df36c2ed-5532-4247-872c-54d13d7506a3-maas-api-xks-ca\") pod \"df36c2ed-5532-4247-872c-54d13d7506a3\" (UID: \"df36c2ed-5532-4247-872c-54d13d7506a3\") " Oct 31 19:35:07.937464 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:07.937447 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df36c2ed-5532-4247-872c-54d13d7506a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df36c2ed-5532-4247-872c-54d13d7506a3-maas-api-openshift-ca\") pod \"df36c2ed-5532-4247-872c-54d13d7506a3\" (UID: \"df36c2ed-5532-4247-872c-54d13d7506a3\") " Oct 31 19:35:07.937510 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:07.937496 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df36c2ed-5532-4247-872c-54d13d7506a3-kube-api-access-mwwj6\" (UniqueName: \"kubernetes.io/projected/df36c2ed-5532-4247-872c-54d13d7506a3-kube-api-access-mwwj6\") pod \"df36c2ed-5532-4247-872c-54d13d7506a3\" (UID: \"df36c2ed-5532-4247-872c-54d13d7506a3\") " Oct 31 19:35:07.937879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:07.937846 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df36c2ed-5532-4247-872c-54d13d7506a3-maas-api-openshift-ca" pod "df36c2ed-5532-4247-872c-54d13d7506a3" (UID: "df36c2ed-5532-4247-872c-54d13d7506a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:35:07.938343 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:07.938322 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df36c2ed-5532-4247-872c-54d13d7506a3-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:07.939497 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:07.939472 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df36c2ed-5532-4247-872c-54d13d7506a3-maas-api-xks-ca" pod "df36c2ed-5532-4247-872c-54d13d7506a3" (UID: "df36c2ed-5532-4247-872c-54d13d7506a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:35:07.939678 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:07.939656 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df36c2ed-5532-4247-872c-54d13d7506a3-kube-api-access-mwwj6" pod "df36c2ed-5532-4247-872c-54d13d7506a3" (UID: "df36c2ed-5532-4247-872c-54d13d7506a3"). InnerVolumeSpecName "kube-api-access-mwwj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:35:08.040662 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:08.040582 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwwj6\" (UniqueName: \"kubernetes.io/projected/df36c2ed-5532-4247-872c-54d13d7506a3-kube-api-access-mwwj6\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:08.040662 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:08.040614 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df36c2ed-5532-4247-872c-54d13d7506a3-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:08.736112 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:08.736078 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7e2625-mgctc" Oct 31 19:35:08.736571 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:08.736114 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e7e2625-mgctc" event={"ID":"df36c2ed-5532-4247-872c-54d13d7506a3","Type":"ContainerDied","Data":"3c1531ff102ae5de022a439a21221c103ad80389813d0775ee18ef42fe7913d6"} Oct 31 19:35:08.736571 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:08.736173 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c1531ff102ae5de022a439a21221c103ad80389813d0775ee18ef42fe7913d6" Oct 31 19:35:08.941857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:08.941829 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2" Oct 31 19:35:08.957550 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:08.957514 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82722265-59cd-40d4-96ad-5cbf775a9cdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82722265-59cd-40d4-96ad-5cbf775a9cdd-maas-api-xks-ca\") pod \"82722265-59cd-40d4-96ad-5cbf775a9cdd\" (UID: \"82722265-59cd-40d4-96ad-5cbf775a9cdd\") " Oct 31 19:35:08.957702 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:08.957623 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82722265-59cd-40d4-96ad-5cbf775a9cdd-kube-api-access-g26zm\" (UniqueName: \"kubernetes.io/projected/82722265-59cd-40d4-96ad-5cbf775a9cdd-kube-api-access-g26zm\") pod \"82722265-59cd-40d4-96ad-5cbf775a9cdd\" (UID: \"82722265-59cd-40d4-96ad-5cbf775a9cdd\") " Oct 31 19:35:08.957799 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:08.957781 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82722265-59cd-40d4-96ad-5cbf775a9cdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82722265-59cd-40d4-96ad-5cbf775a9cdd-maas-api-openshift-ca\") pod \"82722265-59cd-40d4-96ad-5cbf775a9cdd\" (UID: \"82722265-59cd-40d4-96ad-5cbf775a9cdd\") " Oct 31 19:35:08.958228 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:08.958193 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82722265-59cd-40d4-96ad-5cbf775a9cdd-maas-api-openshift-ca" pod "82722265-59cd-40d4-96ad-5cbf775a9cdd" (UID: "82722265-59cd-40d4-96ad-5cbf775a9cdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:35:08.958606 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:08.958583 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82722265-59cd-40d4-96ad-5cbf775a9cdd-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:08.959753 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:08.959725 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82722265-59cd-40d4-96ad-5cbf775a9cdd-maas-api-xks-ca" pod "82722265-59cd-40d4-96ad-5cbf775a9cdd" (UID: "82722265-59cd-40d4-96ad-5cbf775a9cdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:35:08.960340 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:08.960310 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82722265-59cd-40d4-96ad-5cbf775a9cdd-kube-api-access-g26zm" pod "82722265-59cd-40d4-96ad-5cbf775a9cdd" (UID: "82722265-59cd-40d4-96ad-5cbf775a9cdd"). InnerVolumeSpecName "kube-api-access-g26zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:35:09.060214 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:09.060106 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82722265-59cd-40d4-96ad-5cbf775a9cdd-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:09.060214 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:09.060152 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g26zm\" (UniqueName: \"kubernetes.io/projected/82722265-59cd-40d4-96ad-5cbf775a9cdd-kube-api-access-g26zm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:09.746190 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:09.746156 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2" event={"ID":"82722265-59cd-40d4-96ad-5cbf775a9cdd","Type":"ContainerDied","Data":"ac6fee8c6e30cea624d6b14d7359f86ba14a7889c43499ebcf4de87da5cd679e"} Oct 31 19:35:09.746590 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:09.746203 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2" Oct 31 19:35:09.746590 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:09.746230 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac6fee8c6e30cea624d6b14d7359f86ba14a7889c43499ebcf4de87da5cd679e" Oct 31 19:35:12.915398 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:35:12.915363 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:35:12.992217 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:12.992188 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp"] Oct 31 19:35:13.000095 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:13.000057 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-85be723d-kjpmp"] Oct 31 19:35:13.757078 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:13.757041 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="692d071f-24ea-4db2-a615-a7e4f4b85631" path="/var/lib/kubelet/pods/692d071f-24ea-4db2-a615-a7e4f4b85631/volumes" Oct 31 19:35:16.522944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:16.519969 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl"] Oct 31 19:35:16.566934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:16.565250 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0303f0d-q8vbl"] Oct 31 19:35:17.082246 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:17.082211 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh"] Oct 31 19:35:17.082501 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:17.082469 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" podUID="26fe161a-441f-418c-8c6b-1b108999ca7d" containerName="istio-proxy" containerID="cri-o://20d854492ed778f2a72d2b68b35e50f0fb78f37264fe0a23284f4928258e2743" gracePeriod=30 Oct 31 19:35:17.759353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:17.759311 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2d68baa-fee1-441b-8245-cd0083ea6d3a" path="/var/lib/kubelet/pods/e2d68baa-fee1-441b-8245-cd0083ea6d3a/volumes" Oct 31 19:35:22.063023 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:22.062981 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2"] Oct 31 19:35:22.105250 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:22.105196 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1cc6560c-2vzz2"] Oct 31 19:35:22.673939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:22.647265 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:22.673939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:22.647867 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:22.708952 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:22.675978 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:22.831939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:22.828196 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:22.831939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:22.830671 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7"] Oct 31 19:35:22.834027 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:22.833990 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" podUID="099d2ac1-f9dc-4b27-b75e-b13b45b22148" containerName="istio-proxy" containerID="cri-o://59c5bc1fd06f676f1098a0dcfc8f5547d39af133abff507c8e549ec3ae90f231" gracePeriod=30 Oct 31 19:35:22.871453 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:22.871066 4910 generic.go:356] "Generic (PLEG): container finished" podID="26fe161a-441f-418c-8c6b-1b108999ca7d" containerID="20d854492ed778f2a72d2b68b35e50f0fb78f37264fe0a23284f4928258e2743" exitCode=0 Oct 31 19:35:22.871453 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:22.871370 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" event={"ID":"26fe161a-441f-418c-8c6b-1b108999ca7d","Type":"ContainerDied","Data":"20d854492ed778f2a72d2b68b35e50f0fb78f37264fe0a23284f4928258e2743"} Oct 31 19:35:23.253374 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:23.253273 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:23.317508 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:23.317450 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:23.342887 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:23.340993 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:35:23.440480 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:23.440428 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:23.442219 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:23.440732 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:23.443337 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:23.442076 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:23.596695 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:23.596641 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:35:23.619222 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:23.572142 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:23.648179 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:23.648101 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:23.653949 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:23.648855 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:23.663803 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:23.663774 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:35:23.664118 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:23.664100 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:35:23.717799 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:23.717745 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:23.734891 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:23.732026 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:35:23.792040 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:23.782355 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:35:23.834896 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:23.834849 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:35:24.425338 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:24.358126 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:35:24.425338 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:24.359067 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:35:24.425338 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:24.417270 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:24.475282 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:24.466240 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:24.671938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:24.660222 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:24.705172 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:24.705052 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 19:35:24.800961 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:24.781276 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:25.005090 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.005046 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:25.008530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.008487 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:25.009597 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.008663 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:25.013422 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.013381 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:35:25.036942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.014838 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:35:25.158189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.119193 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:25.158189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.144964 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82722265-59cd-40d4-96ad-5cbf775a9cdd" path="/var/lib/kubelet/pods/82722265-59cd-40d4-96ad-5cbf775a9cdd/volumes" Oct 31 19:35:25.158189 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:35:25.145569 4910 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.359s" Oct 31 19:35:25.158189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.145684 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:35:25.158189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.145758 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:35:25.158189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.145816 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:35:25.158189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.145869 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:35:25.158189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.145938 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:35:25.158189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.145996 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:35:25.158189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.146047 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:35:25.158189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.146100 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:35:25.158189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.146150 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:35:25.240370 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.211472 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:35:25.240370 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.239982 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:25.242391 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.242351 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded" Oct 31 19:35:25.427415 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.414526 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:25.429650 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.429588 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:35:25.490990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.482303 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:25.527933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.522134 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:35:25.663346 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.663291 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:35:25.782286 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.778708 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:25.782286 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:25.782007 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:26.025937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.014662 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:26.038300 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.038195 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:26.040019 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.039976 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:35:26.099003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.097441 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26fe161a-441f-418c-8c6b-1b108999ca7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26fe161a-441f-418c-8c6b-1b108999ca7d-istiod-ca-cert\") pod \"26fe161a-441f-418c-8c6b-1b108999ca7d\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " Oct 31 19:35:26.099003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.097512 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-credential-socket\") pod \"26fe161a-441f-418c-8c6b-1b108999ca7d\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " Oct 31 19:35:26.099003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.097554 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-workload-certs\") pod \"26fe161a-441f-418c-8c6b-1b108999ca7d\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " Oct 31 19:35:26.099003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.097615 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-podinfo\") pod \"26fe161a-441f-418c-8c6b-1b108999ca7d\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " Oct 31 19:35:26.099003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.097752 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-data\") pod \"26fe161a-441f-418c-8c6b-1b108999ca7d\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " Oct 31 19:35:26.099003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.097821 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-workload-socket\") pod \"26fe161a-441f-418c-8c6b-1b108999ca7d\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " Oct 31 19:35:26.238501 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.205025 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-credential-socket" pod "26fe161a-441f-418c-8c6b-1b108999ca7d" (UID: "26fe161a-441f-418c-8c6b-1b108999ca7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:35:26.238501 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.230535 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq" podUID="7d56bb36-ed0e-4127-9461-cbec40a46b95" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.51:50052\" within 1s: context deadline exceeded" Oct 31 19:35:26.239058 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.239009 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:35:26.261943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.239547 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:26.262445 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.250351 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26fe161a-441f-418c-8c6b-1b108999ca7d-istiod-ca-cert" pod "26fe161a-441f-418c-8c6b-1b108999ca7d" (UID: "26fe161a-441f-418c-8c6b-1b108999ca7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:35:26.262602 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.254100 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-workload-socket" pod "26fe161a-441f-418c-8c6b-1b108999ca7d" (UID: "26fe161a-441f-418c-8c6b-1b108999ca7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:35:26.262689 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.256564 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-workload-certs" pod "26fe161a-441f-418c-8c6b-1b108999ca7d" (UID: "26fe161a-441f-418c-8c6b-1b108999ca7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:35:26.262779 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.250390 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:26.269113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.253398 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26fe161a-441f-418c-8c6b-1b108999ca7d-kube-api-access-gfcsn\" (UniqueName: \"kubernetes.io/projected/26fe161a-441f-418c-8c6b-1b108999ca7d-kube-api-access-gfcsn\") pod \"26fe161a-441f-418c-8c6b-1b108999ca7d\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " Oct 31 19:35:26.273041 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.273008 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-envoy\") pod \"26fe161a-441f-418c-8c6b-1b108999ca7d\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " Oct 31 19:35:26.277226 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.277200 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-token\" (UniqueName: \"kubernetes.io/projected/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-token\") pod \"26fe161a-441f-418c-8c6b-1b108999ca7d\" (UID: \"26fe161a-441f-418c-8c6b-1b108999ca7d\") " Oct 31 19:35:26.333126 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.333084 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-data" pod "26fe161a-441f-418c-8c6b-1b108999ca7d" (UID: "26fe161a-441f-418c-8c6b-1b108999ca7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:35:26.336395 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.279641 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:26.448937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.435017 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26fe161a-441f-418c-8c6b-1b108999ca7d-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:26.452426 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.452402 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:26.452642 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.452625 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:26.452752 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.452740 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:26.452864 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.452852 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:26.503347 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.503298 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:26.569637 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.568519 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:26.653208 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.653167 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26fe161a-441f-418c-8c6b-1b108999ca7d-kube-api-access-gfcsn" pod "26fe161a-441f-418c-8c6b-1b108999ca7d" (UID: "26fe161a-441f-418c-8c6b-1b108999ca7d"). InnerVolumeSpecName "kube-api-access-gfcsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:35:26.688697 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.670147 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-token" pod "26fe161a-441f-418c-8c6b-1b108999ca7d" (UID: "26fe161a-441f-418c-8c6b-1b108999ca7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:35:26.701936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.682313 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:26.760939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.705718 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfcsn\" (UniqueName: \"kubernetes.io/projected/26fe161a-441f-418c-8c6b-1b108999ca7d-kube-api-access-gfcsn\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:27.105936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.068811 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:27.105936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.793443 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 19:35:27.105936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.776080 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-envoy" pod "26fe161a-441f-418c-8c6b-1b108999ca7d" (UID: "26fe161a-441f-418c-8c6b-1b108999ca7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:35:27.105936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.046214 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-podinfo" pod "26fe161a-441f-418c-8c6b-1b108999ca7d" (UID: "26fe161a-441f-418c-8c6b-1b108999ca7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:35:27.105936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.045859 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:27.105936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:26.815626 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:27.105936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.045021 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:35:27.105936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.045901 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:27.153938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.151539 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:27.192995 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.172764 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:27.194791 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.194762 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26fe161a-441f-418c-8c6b-1b108999ca7d-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:27.239013 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:35:27.238976 4910 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.489s" Oct 31 19:35:27.239285 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.239249 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" event={"ID":"26fe161a-441f-418c-8c6b-1b108999ca7d","Type":"ContainerDied","Data":"8715311ee63f844a75b2472354e036105d0525616862bd85beca04f35837d891"} Oct 31 19:35:27.239647 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.239618 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:35:27.239804 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.239790 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:35:27.239984 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.239968 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:35:27.241202 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.241183 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh" Oct 31 19:35:27.292939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.251212 4910 scope.go:122] "RemoveContainer" containerID="20d854492ed778f2a72d2b68b35e50f0fb78f37264fe0a23284f4928258e2743" Oct 31 19:35:27.319196 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.278574 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded" Oct 31 19:35:27.319572 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.278636 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:35:27.320029 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.278795 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:35:27.321750 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.318008 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:35:27.321950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.321933 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:35:27.323965 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.323938 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:27.345390 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.345168 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:27.441026 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.440985 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:27.546021 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.544996 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:27.707123 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.707072 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:27.730382 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.729290 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:35:27.762496 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.758519 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:35:27.890759 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:27.890722 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:35:28.211837 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:28.211743 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:28.222790 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:28.222754 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:35:28.228877 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:28.228838 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:28.230121 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:28.230008 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:28.265461 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:28.264568 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:35:28.317073 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:28.277975 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-w7w5h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:35:28.317073 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:28.278040 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" podUID="f5b43015-708e-4ef2-883e-cc121fe1c8e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:28.317073 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:28.301692 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:28.457940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:28.409608 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:28.496754 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:28.416287 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:35:28.568873 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:28.549060 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:28.599675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:28.583988 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:28.628586 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:28.628550 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:35:28.715936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:28.713328 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:35:28.971936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:28.954472 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh"] Oct 31 19:35:28.971936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:28.966062 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f0303f0d-openshift-default-5b5588d58f-c5sdh"] Oct 31 19:35:29.115930 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:29.110538 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:35:29.202972 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:29.202749 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:35:29.254045 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:29.235009 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:35:29.254045 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:29.235499 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:35:29.259441 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:29.259398 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:29.341353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:29.317549 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:35:29.392933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:29.385012 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:29.497606 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:29.497465 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:29.563028 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:29.561742 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:35:29.581031 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:29.580998 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:35:29.617661 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:29.617330 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:35:29.648936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:29.648557 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:35:29.867069 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:29.866984 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26fe161a-441f-418c-8c6b-1b108999ca7d" path="/var/lib/kubelet/pods/26fe161a-441f-418c-8c6b-1b108999ca7d/volumes" Oct 31 19:35:29.867600 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:29.867564 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:35:29.867811 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:29.867798 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:35:29.916185 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:29.916149 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:35:34.810445 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:34.810409 4910 patch_prober.go:29] interesting pod/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:35:34.811082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:34.810462 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" podUID="099d2ac1-f9dc-4b27-b75e-b13b45b22148" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:36.381533 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.381506 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:35:36.544423 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.544330 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-workload-socket\") pod \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " Oct 31 19:35:36.544606 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.544441 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-credential-socket\") pod \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " Oct 31 19:35:36.544606 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.544491 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/099d2ac1-f9dc-4b27-b75e-b13b45b22148-kube-api-access-bj48w\" (UniqueName: \"kubernetes.io/projected/099d2ac1-f9dc-4b27-b75e-b13b45b22148-kube-api-access-bj48w\") pod \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " Oct 31 19:35:36.544606 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.544517 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-podinfo\") pod \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " Oct 31 19:35:36.544606 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.544582 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-envoy\") pod \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " Oct 31 19:35:36.544845 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.544657 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-workload-certs\") pod \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " Oct 31 19:35:36.544845 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.544719 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-token\" (UniqueName: \"kubernetes.io/projected/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-token\") pod \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " Oct 31 19:35:36.544845 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.544727 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-workload-socket" pod "099d2ac1-f9dc-4b27-b75e-b13b45b22148" (UID: "099d2ac1-f9dc-4b27-b75e-b13b45b22148"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:35:36.544845 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.544757 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istiod-ca-cert\") pod \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " Oct 31 19:35:36.544845 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.544778 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-credential-socket" pod "099d2ac1-f9dc-4b27-b75e-b13b45b22148" (UID: "099d2ac1-f9dc-4b27-b75e-b13b45b22148"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:35:36.545121 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.544951 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-data\") pod \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\" (UID: \"099d2ac1-f9dc-4b27-b75e-b13b45b22148\") " Oct 31 19:35:36.545121 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.544976 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-workload-certs" pod "099d2ac1-f9dc-4b27-b75e-b13b45b22148" (UID: "099d2ac1-f9dc-4b27-b75e-b13b45b22148"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:35:36.545265 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.545247 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-data" pod "099d2ac1-f9dc-4b27-b75e-b13b45b22148" (UID: "099d2ac1-f9dc-4b27-b75e-b13b45b22148"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:35:36.545359 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.545334 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istiod-ca-cert" pod "099d2ac1-f9dc-4b27-b75e-b13b45b22148" (UID: "099d2ac1-f9dc-4b27-b75e-b13b45b22148"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:35:36.545861 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.545839 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:36.545922 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.545866 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:36.545922 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.545882 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:36.545922 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.545896 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:36.546049 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.545923 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:36.547133 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.547104 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-envoy" pod "099d2ac1-f9dc-4b27-b75e-b13b45b22148" (UID: "099d2ac1-f9dc-4b27-b75e-b13b45b22148"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:35:36.547455 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.547427 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-token" pod "099d2ac1-f9dc-4b27-b75e-b13b45b22148" (UID: "099d2ac1-f9dc-4b27-b75e-b13b45b22148"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:35:36.547744 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.547725 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-podinfo" pod "099d2ac1-f9dc-4b27-b75e-b13b45b22148" (UID: "099d2ac1-f9dc-4b27-b75e-b13b45b22148"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:35:36.548597 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.548573 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099d2ac1-f9dc-4b27-b75e-b13b45b22148-kube-api-access-bj48w" pod "099d2ac1-f9dc-4b27-b75e-b13b45b22148" (UID: "099d2ac1-f9dc-4b27-b75e-b13b45b22148"). InnerVolumeSpecName "kube-api-access-bj48w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:35:36.648302 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.648255 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj48w\" (UniqueName: \"kubernetes.io/projected/099d2ac1-f9dc-4b27-b75e-b13b45b22148-kube-api-access-bj48w\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:36.648302 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.648288 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:36.648302 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.648302 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:36.648578 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.648313 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099d2ac1-f9dc-4b27-b75e-b13b45b22148-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:36.710443 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.710410 4910 generic.go:356] "Generic (PLEG): container finished" podID="099d2ac1-f9dc-4b27-b75e-b13b45b22148" containerID="59c5bc1fd06f676f1098a0dcfc8f5547d39af133abff507c8e549ec3ae90f231" exitCode=0 Oct 31 19:35:36.710635 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.710459 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" Oct 31 19:35:36.710635 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.710492 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" event={"ID":"099d2ac1-f9dc-4b27-b75e-b13b45b22148","Type":"ContainerDied","Data":"59c5bc1fd06f676f1098a0dcfc8f5547d39af133abff507c8e549ec3ae90f231"} Oct 31 19:35:36.710635 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.710547 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7" event={"ID":"099d2ac1-f9dc-4b27-b75e-b13b45b22148","Type":"ContainerDied","Data":"15f20b2e0a6abf46ddca65b943590fe942c32cdbfd50c21e957e0971e1feb72a"} Oct 31 19:35:36.710635 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.710573 4910 scope.go:122] "RemoveContainer" containerID="59c5bc1fd06f676f1098a0dcfc8f5547d39af133abff507c8e549ec3ae90f231" Oct 31 19:35:36.724127 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.724105 4910 scope.go:122] "RemoveContainer" containerID="59c5bc1fd06f676f1098a0dcfc8f5547d39af133abff507c8e549ec3ae90f231" Oct 31 19:35:36.724425 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:35:36.724402 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59c5bc1fd06f676f1098a0dcfc8f5547d39af133abff507c8e549ec3ae90f231\": container with ID starting with 59c5bc1fd06f676f1098a0dcfc8f5547d39af133abff507c8e549ec3ae90f231 not found: ID does not exist" containerID="59c5bc1fd06f676f1098a0dcfc8f5547d39af133abff507c8e549ec3ae90f231" Oct 31 19:35:36.724507 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.724434 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59c5bc1fd06f676f1098a0dcfc8f5547d39af133abff507c8e549ec3ae90f231"} err="failed to get container status \"59c5bc1fd06f676f1098a0dcfc8f5547d39af133abff507c8e549ec3ae90f231\": rpc error: code = NotFound desc = could not find container \"59c5bc1fd06f676f1098a0dcfc8f5547d39af133abff507c8e549ec3ae90f231\": container with ID starting with 59c5bc1fd06f676f1098a0dcfc8f5547d39af133abff507c8e549ec3ae90f231 not found: ID does not exist" Oct 31 19:35:36.754782 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.754745 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7"] Oct 31 19:35:36.766736 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:36.766694 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1cc6560c-openshift-default-c5d595b6c-7mpf7"] Oct 31 19:35:37.757661 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:37.757623 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="099d2ac1-f9dc-4b27-b75e-b13b45b22148" path="/var/lib/kubelet/pods/099d2ac1-f9dc-4b27-b75e-b13b45b22148/volumes" Oct 31 19:35:42.714621 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:35:42.714584 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:35:44.068016 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.067962 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e248b878-72fgf"] Oct 31 19:35:44.074704 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.074673 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e248b878-72fgf"] Oct 31 19:35:44.367297 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.367215 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63eab93e-drfqz"] Oct 31 19:35:44.367971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.367954 4910 cpu_manager.go:414] "containerMap: removing container" podUID="df36c2ed-5532-4247-872c-54d13d7506a3" containerName="revoke-keys" Oct 31 19:35:44.368029 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.367973 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df36c2ed-5532-4247-872c-54d13d7506a3" containerName="revoke-keys" Oct 31 19:35:44.368029 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.367998 4910 cpu_manager.go:414] "containerMap: removing container" podUID="099d2ac1-f9dc-4b27-b75e-b13b45b22148" containerName="istio-proxy" Oct 31 19:35:44.368029 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.368004 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="099d2ac1-f9dc-4b27-b75e-b13b45b22148" containerName="istio-proxy" Oct 31 19:35:44.368029 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.368027 4910 cpu_manager.go:414] "containerMap: removing container" podUID="82722265-59cd-40d4-96ad-5cbf775a9cdd" containerName="revoke-keys" Oct 31 19:35:44.368157 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.368033 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82722265-59cd-40d4-96ad-5cbf775a9cdd" containerName="revoke-keys" Oct 31 19:35:44.368157 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.368057 4910 cpu_manager.go:414] "containerMap: removing container" podUID="26fe161a-441f-418c-8c6b-1b108999ca7d" containerName="istio-proxy" Oct 31 19:35:44.368157 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.368062 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26fe161a-441f-418c-8c6b-1b108999ca7d" containerName="istio-proxy" Oct 31 19:35:44.368250 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.368173 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="26fe161a-441f-418c-8c6b-1b108999ca7d" containerName="istio-proxy" Oct 31 19:35:44.368250 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.368181 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="df36c2ed-5532-4247-872c-54d13d7506a3" containerName="revoke-keys" Oct 31 19:35:44.368250 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.368191 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="099d2ac1-f9dc-4b27-b75e-b13b45b22148" containerName="istio-proxy" Oct 31 19:35:44.368250 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.368201 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="82722265-59cd-40d4-96ad-5cbf775a9cdd" containerName="revoke-keys" Oct 31 19:35:44.372518 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.372500 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63eab93e-drfqz" Oct 31 19:35:44.377796 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.377761 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:35:44.377981 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.377807 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:35:44.384758 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.384733 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63eab93e-drfqz"] Oct 31 19:35:44.436790 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.436751 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg5lb\" (UniqueName: \"kubernetes.io/projected/b20001d2-c0ae-46c3-b0bc-5fab010f3a9b-kube-api-access-gg5lb\") pod \"maas-api-revoke-keys-e2e-ait-63eab93e-drfqz\" (UID: \"b20001d2-c0ae-46c3-b0bc-5fab010f3a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63eab93e-drfqz" Oct 31 19:35:44.437044 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.436837 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b20001d2-c0ae-46c3-b0bc-5fab010f3a9b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-63eab93e-drfqz\" (UID: \"b20001d2-c0ae-46c3-b0bc-5fab010f3a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63eab93e-drfqz" Oct 31 19:35:44.437044 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.436985 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b20001d2-c0ae-46c3-b0bc-5fab010f3a9b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-63eab93e-drfqz\" (UID: \"b20001d2-c0ae-46c3-b0bc-5fab010f3a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63eab93e-drfqz" Oct 31 19:35:44.538578 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.538540 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5lb\" (UniqueName: \"kubernetes.io/projected/b20001d2-c0ae-46c3-b0bc-5fab010f3a9b-kube-api-access-gg5lb\") pod \"maas-api-revoke-keys-e2e-ait-63eab93e-drfqz\" (UID: \"b20001d2-c0ae-46c3-b0bc-5fab010f3a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63eab93e-drfqz" Oct 31 19:35:44.538768 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.538617 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b20001d2-c0ae-46c3-b0bc-5fab010f3a9b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-63eab93e-drfqz\" (UID: \"b20001d2-c0ae-46c3-b0bc-5fab010f3a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63eab93e-drfqz" Oct 31 19:35:44.538768 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.538665 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b20001d2-c0ae-46c3-b0bc-5fab010f3a9b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-63eab93e-drfqz\" (UID: \"b20001d2-c0ae-46c3-b0bc-5fab010f3a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63eab93e-drfqz" Oct 31 19:35:44.539546 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.539514 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b20001d2-c0ae-46c3-b0bc-5fab010f3a9b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-63eab93e-drfqz\" (UID: \"b20001d2-c0ae-46c3-b0bc-5fab010f3a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63eab93e-drfqz" Oct 31 19:35:44.541207 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.541179 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b20001d2-c0ae-46c3-b0bc-5fab010f3a9b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-63eab93e-drfqz\" (UID: \"b20001d2-c0ae-46c3-b0bc-5fab010f3a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63eab93e-drfqz" Oct 31 19:35:44.551205 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.551177 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg5lb\" (UniqueName: \"kubernetes.io/projected/b20001d2-c0ae-46c3-b0bc-5fab010f3a9b-kube-api-access-gg5lb\") pod \"maas-api-revoke-keys-e2e-ait-63eab93e-drfqz\" (UID: \"b20001d2-c0ae-46c3-b0bc-5fab010f3a9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63eab93e-drfqz" Oct 31 19:35:44.695219 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:44.695134 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63eab93e-drfqz" Oct 31 19:35:45.146924 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:45.146885 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63eab93e-drfqz"] Oct 31 19:35:45.757577 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:45.757499 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d941864a-96b8-4460-8fff-23abdc637c19" path="/var/lib/kubelet/pods/d941864a-96b8-4460-8fff-23abdc637c19/volumes" Oct 31 19:35:45.804063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:45.804027 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63eab93e-drfqz" event={"ID":"b20001d2-c0ae-46c3-b0bc-5fab010f3a9b","Type":"ContainerStarted","Data":"30e21a4855c8be96cb1e0e5db2b175df59848fc5f3e669793624eaf492d64936"} Oct 31 19:35:45.804259 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:45.804080 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63eab93e-drfqz" event={"ID":"b20001d2-c0ae-46c3-b0bc-5fab010f3a9b","Type":"ContainerStarted","Data":"d4f035c1be27d53c01ec182ea0534f447bd22f1ea04c889bae7ebab39fd1b39b"} Oct 31 19:35:46.814802 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:46.814762 4910 generic.go:356] "Generic (PLEG): container finished" podID="b20001d2-c0ae-46c3-b0bc-5fab010f3a9b" containerID="30e21a4855c8be96cb1e0e5db2b175df59848fc5f3e669793624eaf492d64936" exitCode=0 Oct 31 19:35:46.815179 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:46.814846 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63eab93e-drfqz" event={"ID":"b20001d2-c0ae-46c3-b0bc-5fab010f3a9b","Type":"ContainerDied","Data":"30e21a4855c8be96cb1e0e5db2b175df59848fc5f3e669793624eaf492d64936"} Oct 31 19:35:46.949674 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:46.949636 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6"] Oct 31 19:35:46.954645 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:46.954618 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6" Oct 31 19:35:46.970085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:46.970050 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6"] Oct 31 19:35:47.139408 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:47.139317 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ad5ecb-de74-49e7-a9bb-d84890332c73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6\" (UID: \"60ad5ecb-de74-49e7-a9bb-d84890332c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6" Oct 31 19:35:47.139558 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:47.139510 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvh8n\" (UniqueName: \"kubernetes.io/projected/60ad5ecb-de74-49e7-a9bb-d84890332c73-kube-api-access-nvh8n\") pod \"maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6\" (UID: \"60ad5ecb-de74-49e7-a9bb-d84890332c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6" Oct 31 19:35:47.139600 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:47.139577 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ad5ecb-de74-49e7-a9bb-d84890332c73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6\" (UID: \"60ad5ecb-de74-49e7-a9bb-d84890332c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6" Oct 31 19:35:47.242932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:47.241289 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ad5ecb-de74-49e7-a9bb-d84890332c73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6\" (UID: \"60ad5ecb-de74-49e7-a9bb-d84890332c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6" Oct 31 19:35:47.242932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:47.241422 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvh8n\" (UniqueName: \"kubernetes.io/projected/60ad5ecb-de74-49e7-a9bb-d84890332c73-kube-api-access-nvh8n\") pod \"maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6\" (UID: \"60ad5ecb-de74-49e7-a9bb-d84890332c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6" Oct 31 19:35:47.242932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:47.241466 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ad5ecb-de74-49e7-a9bb-d84890332c73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6\" (UID: \"60ad5ecb-de74-49e7-a9bb-d84890332c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6" Oct 31 19:35:47.242932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:47.242329 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ad5ecb-de74-49e7-a9bb-d84890332c73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6\" (UID: \"60ad5ecb-de74-49e7-a9bb-d84890332c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6" Oct 31 19:35:47.245431 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:47.245401 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ad5ecb-de74-49e7-a9bb-d84890332c73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6\" (UID: \"60ad5ecb-de74-49e7-a9bb-d84890332c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6" Oct 31 19:35:47.256482 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:47.256452 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvh8n\" (UniqueName: \"kubernetes.io/projected/60ad5ecb-de74-49e7-a9bb-d84890332c73-kube-api-access-nvh8n\") pod \"maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6\" (UID: \"60ad5ecb-de74-49e7-a9bb-d84890332c73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6" Oct 31 19:35:47.275162 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:47.275132 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6" Oct 31 19:35:47.564229 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:35:47.564195 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ad5ecb_de74_49e7_a9bb_d84890332c73.slice/crio-68ad7763b7dd7a7d4047eac714c296b1b4d007a9303ee16dfc281e6129a0f493 WatchSource:0}: Error finding container 68ad7763b7dd7a7d4047eac714c296b1b4d007a9303ee16dfc281e6129a0f493: Status 404 returned error can't find the container with id 68ad7763b7dd7a7d4047eac714c296b1b4d007a9303ee16dfc281e6129a0f493 Oct 31 19:35:47.569360 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:47.569285 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6"] Oct 31 19:35:47.826952 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:47.826831 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6" event={"ID":"60ad5ecb-de74-49e7-a9bb-d84890332c73","Type":"ContainerStarted","Data":"68ad7763b7dd7a7d4047eac714c296b1b4d007a9303ee16dfc281e6129a0f493"} Oct 31 19:35:48.105731 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:48.105700 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63eab93e-drfqz" Oct 31 19:35:48.157812 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:48.157773 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b20001d2-c0ae-46c3-b0bc-5fab010f3a9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b20001d2-c0ae-46c3-b0bc-5fab010f3a9b-maas-api-openshift-ca\") pod \"b20001d2-c0ae-46c3-b0bc-5fab010f3a9b\" (UID: \"b20001d2-c0ae-46c3-b0bc-5fab010f3a9b\") " Oct 31 19:35:48.158045 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:48.157936 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b20001d2-c0ae-46c3-b0bc-5fab010f3a9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b20001d2-c0ae-46c3-b0bc-5fab010f3a9b-maas-api-xks-ca\") pod \"b20001d2-c0ae-46c3-b0bc-5fab010f3a9b\" (UID: \"b20001d2-c0ae-46c3-b0bc-5fab010f3a9b\") " Oct 31 19:35:48.158116 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:48.158051 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b20001d2-c0ae-46c3-b0bc-5fab010f3a9b-kube-api-access-gg5lb\" (UniqueName: \"kubernetes.io/projected/b20001d2-c0ae-46c3-b0bc-5fab010f3a9b-kube-api-access-gg5lb\") pod \"b20001d2-c0ae-46c3-b0bc-5fab010f3a9b\" (UID: \"b20001d2-c0ae-46c3-b0bc-5fab010f3a9b\") " Oct 31 19:35:48.159341 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:48.159313 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b20001d2-c0ae-46c3-b0bc-5fab010f3a9b-maas-api-openshift-ca" pod "b20001d2-c0ae-46c3-b0bc-5fab010f3a9b" (UID: "b20001d2-c0ae-46c3-b0bc-5fab010f3a9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:35:48.160880 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:48.160851 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b20001d2-c0ae-46c3-b0bc-5fab010f3a9b-maas-api-xks-ca" pod "b20001d2-c0ae-46c3-b0bc-5fab010f3a9b" (UID: "b20001d2-c0ae-46c3-b0bc-5fab010f3a9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:35:48.161290 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:48.161259 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b20001d2-c0ae-46c3-b0bc-5fab010f3a9b-kube-api-access-gg5lb" pod "b20001d2-c0ae-46c3-b0bc-5fab010f3a9b" (UID: "b20001d2-c0ae-46c3-b0bc-5fab010f3a9b"). InnerVolumeSpecName "kube-api-access-gg5lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:35:48.260109 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:48.260075 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b20001d2-c0ae-46c3-b0bc-5fab010f3a9b-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:48.260242 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:48.260113 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b20001d2-c0ae-46c3-b0bc-5fab010f3a9b-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:48.260242 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:48.260128 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg5lb\" (UniqueName: \"kubernetes.io/projected/b20001d2-c0ae-46c3-b0bc-5fab010f3a9b-kube-api-access-gg5lb\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:48.841631 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:48.841594 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63eab93e-drfqz" Oct 31 19:35:48.841631 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:48.841622 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63eab93e-drfqz" event={"ID":"b20001d2-c0ae-46c3-b0bc-5fab010f3a9b","Type":"ContainerDied","Data":"d4f035c1be27d53c01ec182ea0534f447bd22f1ea04c889bae7ebab39fd1b39b"} Oct 31 19:35:48.842229 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:48.841673 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4f035c1be27d53c01ec182ea0534f447bd22f1ea04c889bae7ebab39fd1b39b" Oct 31 19:35:48.843840 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:48.843811 4910 generic.go:356] "Generic (PLEG): container finished" podID="60ad5ecb-de74-49e7-a9bb-d84890332c73" containerID="6ebc6bfd83c60d818d2100ed6cfe63fde9445d4960c68b3929217077289a31f3" exitCode=0 Oct 31 19:35:48.843983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:48.843904 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6" event={"ID":"60ad5ecb-de74-49e7-a9bb-d84890332c73","Type":"ContainerDied","Data":"6ebc6bfd83c60d818d2100ed6cfe63fde9445d4960c68b3929217077289a31f3"} Oct 31 19:35:50.026681 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:50.026654 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6" Oct 31 19:35:50.090121 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:50.090086 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ad5ecb-de74-49e7-a9bb-d84890332c73-kube-api-access-nvh8n\" (UniqueName: \"kubernetes.io/projected/60ad5ecb-de74-49e7-a9bb-d84890332c73-kube-api-access-nvh8n\") pod \"60ad5ecb-de74-49e7-a9bb-d84890332c73\" (UID: \"60ad5ecb-de74-49e7-a9bb-d84890332c73\") " Oct 31 19:35:50.090328 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:50.090231 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60ad5ecb-de74-49e7-a9bb-d84890332c73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ad5ecb-de74-49e7-a9bb-d84890332c73-maas-api-xks-ca\") pod \"60ad5ecb-de74-49e7-a9bb-d84890332c73\" (UID: \"60ad5ecb-de74-49e7-a9bb-d84890332c73\") " Oct 31 19:35:50.090328 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:50.090298 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60ad5ecb-de74-49e7-a9bb-d84890332c73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ad5ecb-de74-49e7-a9bb-d84890332c73-maas-api-openshift-ca\") pod \"60ad5ecb-de74-49e7-a9bb-d84890332c73\" (UID: \"60ad5ecb-de74-49e7-a9bb-d84890332c73\") " Oct 31 19:35:50.090708 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:50.090682 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ad5ecb-de74-49e7-a9bb-d84890332c73-maas-api-openshift-ca" pod "60ad5ecb-de74-49e7-a9bb-d84890332c73" (UID: "60ad5ecb-de74-49e7-a9bb-d84890332c73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:35:50.091064 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:50.091041 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60ad5ecb-de74-49e7-a9bb-d84890332c73-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:50.092412 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:50.092378 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ad5ecb-de74-49e7-a9bb-d84890332c73-maas-api-xks-ca" pod "60ad5ecb-de74-49e7-a9bb-d84890332c73" (UID: "60ad5ecb-de74-49e7-a9bb-d84890332c73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:35:50.092513 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:50.092441 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ad5ecb-de74-49e7-a9bb-d84890332c73-kube-api-access-nvh8n" pod "60ad5ecb-de74-49e7-a9bb-d84890332c73" (UID: "60ad5ecb-de74-49e7-a9bb-d84890332c73"). InnerVolumeSpecName "kube-api-access-nvh8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:35:50.194546 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:50.193080 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvh8n\" (UniqueName: \"kubernetes.io/projected/60ad5ecb-de74-49e7-a9bb-d84890332c73-kube-api-access-nvh8n\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:50.194546 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:50.193121 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60ad5ecb-de74-49e7-a9bb-d84890332c73-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:35:50.868707 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:50.868676 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6" Oct 31 19:35:50.868958 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:50.868704 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6" event={"ID":"60ad5ecb-de74-49e7-a9bb-d84890332c73","Type":"ContainerDied","Data":"68ad7763b7dd7a7d4047eac714c296b1b4d007a9303ee16dfc281e6129a0f493"} Oct 31 19:35:50.868958 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:50.868757 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68ad7763b7dd7a7d4047eac714c296b1b4d007a9303ee16dfc281e6129a0f493" Oct 31 19:35:54.039892 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:54.039853 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d119c4a7-dxl8c"] Oct 31 19:35:54.046416 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:54.046383 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d119c4a7-dxl8c"] Oct 31 19:35:55.760903 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:55.760848 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b07f8f0e-9ef5-4609-9302-39562f5f9074" path="/var/lib/kubelet/pods/b07f8f0e-9ef5-4609-9302-39562f5f9074/volumes" Oct 31 19:35:57.112937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:57.110723 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60a52118-k8fzb"] Oct 31 19:35:57.188936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:57.163522 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60a52118-k8fzb"] Oct 31 19:35:57.604991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:57.602809 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:57.868550 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:57.854030 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:58.088186 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:58.087199 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:35:58.088186 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:58.087660 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:58.128212 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:58.113179 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="044955bb-e4be-49d1-9dfd-5500fb1094d5" path="/var/lib/kubelet/pods/044955bb-e4be-49d1-9dfd-5500fb1094d5/volumes" Oct 31 19:35:58.238932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:58.227195 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:58.354577 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:58.354541 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:35:58.354895 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:58.354860 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" podUID="f9561493-df06-4f24-878e-f5d1255ac41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:58.482547 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:58.482446 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:58.489808 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:58.489773 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:35:58.759002 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:58.751155 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:59.108346 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:59.068360 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:59.108346 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:59.073154 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded" Oct 31 19:35:59.108346 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:59.073318 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:59.108346 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:59.073398 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:59.108346 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:59.088106 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:59.108346 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:59.088472 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:35:59.108346 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:59.088787 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:35:59.108346 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:59.088843 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:35:59.442532 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:59.435032 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:59.515932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:59.460582 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:35:59.744993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:59.471202 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:59.845262 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:59.781288 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:35:59.873934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:35:59.837148 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 19:36:00.012557 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:00.012438 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:36:00.080943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:00.073591 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:36:00.080943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:00.074644 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:36:00.080943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:00.075537 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:36:00.108974 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:00.106032 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:36:00.108974 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:00.106342 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 19:36:00.569858 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:00.569813 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Oct 31 19:36:00.643285 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:00.581277 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:36:00.648189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:00.648158 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:36:00.651032 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:00.651008 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:36:00.653328 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:00.653298 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:36:00.654796 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:00.654764 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:36:00.654967 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:00.654952 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:36:00.655101 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:00.655088 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:36:00.655240 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:00.655227 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:36:00.655341 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:00.641082 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:36:00.846019 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:00.845935 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63eab93e-drfqz"] Oct 31 19:36:00.921932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:00.897989 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:36:00.921932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:00.916028 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:36:00.922434 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:00.922396 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:36:00.922975 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:00.922953 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:36:00.962211 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:00.962177 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-63eab93e-drfqz"] Oct 31 19:36:00.985967 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:00.965596 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:36:00.985967 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:00.966158 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:36:01.072050 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:01.072009 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6qm4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:36:01.072351 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:01.072318 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" podUID="c73a891d-f2de-4099-9a98-bdcbae51888f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:36:01.101008 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:01.082206 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:36:01.101008 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:01.092058 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:36:01.101008 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:01.098025 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:36:01.135110 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:01.114660 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:36:01.165986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:01.165935 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:36:01.510949 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:01.510834 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:36:01.662067 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:01.662022 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:36:01.663162 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:01.663139 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:36:01.740949 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:01.731065 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:36:01.895556 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:01.895517 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b20001d2-c0ae-46c3-b0bc-5fab010f3a9b" path="/var/lib/kubelet/pods/b20001d2-c0ae-46c3-b0bc-5fab010f3a9b/volumes" Oct 31 19:36:01.896432 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:01.896408 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:36:01.929066 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:01.929026 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:36:01.966042 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:01.965993 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:36:02.028043 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:02.027996 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:36:02.097335 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:02.097285 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:36:02.123937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:02.100623 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:36:02.183039 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:02.124223 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:36:02.183039 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:02.151303 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 19:36:02.185192 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:02.181156 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:36:02.185192 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:02.184968 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:36:02.384013 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:02.383723 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:36:02.516845 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:02.516733 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:36:02.517272 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:02.517243 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:36:02.627938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:02.613560 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:36:02.685239 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:02.684450 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:36:02.686685 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:02.686649 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:36:02.687945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:02.687926 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:36:02.723889 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:02.723848 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:36:02.771793 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:02.736980 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:36:02.771793 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:02.737378 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:36:02.771793 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:02.737823 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:36:02.771793 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:02.747429 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:36:02.777332 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:02.777304 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:36:02.908124 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:02.907251 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:36:02.908124 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:02.907849 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:36:02.913226 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:02.913182 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:36:04.781745 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:04.781714 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6"] Oct 31 19:36:04.791657 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:04.791626 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0b9ff685-tvpj6"] Oct 31 19:36:05.775124 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:05.775090 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60ad5ecb-de74-49e7-a9bb-d84890332c73" path="/var/lib/kubelet/pods/60ad5ecb-de74-49e7-a9bb-d84890332c73/volumes" Oct 31 19:36:06.050787 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:06.050670 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fed47bff-kz7g9"] Oct 31 19:36:06.051571 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:06.051539 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fed47bff-kz7g9"] Oct 31 19:36:07.758868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:07.758833 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab24981d-834d-49d0-bfa7-eb64ff0ef2d4" path="/var/lib/kubelet/pods/ab24981d-834d-49d0-bfa7-eb64ff0ef2d4/volumes" Oct 31 19:36:10.047030 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:10.046988 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b891f4f-vnc7m"] Oct 31 19:36:10.050856 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:10.050832 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b891f4f-vnc7m"] Oct 31 19:36:11.758601 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:11.758565 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9869f04-02cf-4d2a-bd99-cf22f64910a8" path="/var/lib/kubelet/pods/c9869f04-02cf-4d2a-bd99-cf22f64910a8/volumes" Oct 31 19:36:12.713621 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:36:12.713580 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:36:16.065924 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:16.061840 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f8d8368-dm6rd"] Oct 31 19:36:16.072047 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:16.072008 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f8d8368-dm6rd"] Oct 31 19:36:17.757750 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:17.757705 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec" path="/var/lib/kubelet/pods/1d99da01-5cc4-44c4-8ba5-3b50abd0d2ec/volumes" Oct 31 19:36:18.042588 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:18.042494 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96cb75ff-9msqn"] Oct 31 19:36:18.051328 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:18.051279 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96cb75ff-9msqn"] Oct 31 19:36:19.759887 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:19.759853 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7" path="/var/lib/kubelet/pods/a7d539c3-a35c-4d49-ae8d-0d1ed5b405b7/volumes" Oct 31 19:36:20.040483 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:20.040394 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9a85867-9nm5s"] Oct 31 19:36:20.046516 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:20.046489 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9a85867-9nm5s"] Oct 31 19:36:21.035279 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:21.035240 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fa6423c-85cc8"] Oct 31 19:36:21.049593 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:21.049558 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fa6423c-85cc8"] Oct 31 19:36:21.758781 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:21.758745 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dd5789d-652e-401f-9ab7-fe1bc7adc002" path="/var/lib/kubelet/pods/3dd5789d-652e-401f-9ab7-fe1bc7adc002/volumes" Oct 31 19:36:21.759404 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:21.759382 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa4000ef-20a4-4566-bbfb-32be1f61145c" path="/var/lib/kubelet/pods/fa4000ef-20a4-4566-bbfb-32be1f61145c/volumes" Oct 31 19:36:29.050326 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:29.050258 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98dd5c4e-xg769"] Oct 31 19:36:29.059401 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:29.059271 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98dd5c4e-xg769"] Oct 31 19:36:29.758999 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:29.758959 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b05a049b-9fe7-4ee0-b007-4149edb2da92" path="/var/lib/kubelet/pods/b05a049b-9fe7-4ee0-b007-4149edb2da92/volumes" Oct 31 19:36:32.043981 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:32.043220 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8ed7d9a-4f6bk"] Oct 31 19:36:32.055275 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:32.055242 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8ed7d9a-4f6bk"] Oct 31 19:36:32.869585 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:32.869550 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b66ef818-fwll4"] Oct 31 19:36:32.870349 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:32.870328 4910 cpu_manager.go:414] "containerMap: removing container" podUID="b20001d2-c0ae-46c3-b0bc-5fab010f3a9b" containerName="revoke-keys" Oct 31 19:36:32.870349 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:32.870346 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b20001d2-c0ae-46c3-b0bc-5fab010f3a9b" containerName="revoke-keys" Oct 31 19:36:32.870530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:32.870374 4910 cpu_manager.go:414] "containerMap: removing container" podUID="60ad5ecb-de74-49e7-a9bb-d84890332c73" containerName="revoke-keys" Oct 31 19:36:32.870530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:32.870381 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ad5ecb-de74-49e7-a9bb-d84890332c73" containerName="revoke-keys" Oct 31 19:36:32.870530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:32.870485 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ad5ecb-de74-49e7-a9bb-d84890332c73" containerName="revoke-keys" Oct 31 19:36:32.870530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:32.870500 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="b20001d2-c0ae-46c3-b0bc-5fab010f3a9b" containerName="revoke-keys" Oct 31 19:36:32.876028 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:32.875997 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66ef818-fwll4" Oct 31 19:36:32.883098 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:32.883063 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:36:32.883238 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:32.883118 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:36:32.886000 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:32.885972 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b66ef818-fwll4"] Oct 31 19:36:32.968243 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:32.968188 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hmct\" (UniqueName: \"kubernetes.io/projected/5ad9721c-495b-413e-b60d-29e34471c03d-kube-api-access-9hmct\") pod \"maas-api-revoke-sub-b66ef818-fwll4\" (UID: \"5ad9721c-495b-413e-b60d-29e34471c03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66ef818-fwll4" Oct 31 19:36:32.968440 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:32.968369 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad9721c-495b-413e-b60d-29e34471c03d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b66ef818-fwll4\" (UID: \"5ad9721c-495b-413e-b60d-29e34471c03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66ef818-fwll4" Oct 31 19:36:32.968527 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:32.968510 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad9721c-495b-413e-b60d-29e34471c03d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b66ef818-fwll4\" (UID: \"5ad9721c-495b-413e-b60d-29e34471c03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66ef818-fwll4" Oct 31 19:36:33.073080 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:33.073038 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hmct\" (UniqueName: \"kubernetes.io/projected/5ad9721c-495b-413e-b60d-29e34471c03d-kube-api-access-9hmct\") pod \"maas-api-revoke-sub-b66ef818-fwll4\" (UID: \"5ad9721c-495b-413e-b60d-29e34471c03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66ef818-fwll4" Oct 31 19:36:33.074217 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:33.074176 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad9721c-495b-413e-b60d-29e34471c03d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b66ef818-fwll4\" (UID: \"5ad9721c-495b-413e-b60d-29e34471c03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66ef818-fwll4" Oct 31 19:36:33.074553 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:33.074532 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad9721c-495b-413e-b60d-29e34471c03d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b66ef818-fwll4\" (UID: \"5ad9721c-495b-413e-b60d-29e34471c03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66ef818-fwll4" Oct 31 19:36:33.081545 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:33.081282 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad9721c-495b-413e-b60d-29e34471c03d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b66ef818-fwll4\" (UID: \"5ad9721c-495b-413e-b60d-29e34471c03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66ef818-fwll4" Oct 31 19:36:33.100464 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:33.100429 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad9721c-495b-413e-b60d-29e34471c03d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b66ef818-fwll4\" (UID: \"5ad9721c-495b-413e-b60d-29e34471c03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66ef818-fwll4" Oct 31 19:36:33.122751 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:33.122677 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hmct\" (UniqueName: \"kubernetes.io/projected/5ad9721c-495b-413e-b60d-29e34471c03d-kube-api-access-9hmct\") pod \"maas-api-revoke-sub-b66ef818-fwll4\" (UID: \"5ad9721c-495b-413e-b60d-29e34471c03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66ef818-fwll4" Oct 31 19:36:33.199282 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:33.199252 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66ef818-fwll4" Oct 31 19:36:33.509627 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:36:33.509590 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad9721c_495b_413e_b60d_29e34471c03d.slice/crio-9b7746c8e2992d6b36dff1265e3fe1a5fc688f7dcc4ce8784805fae49b3fe30f WatchSource:0}: Error finding container 9b7746c8e2992d6b36dff1265e3fe1a5fc688f7dcc4ce8784805fae49b3fe30f: Status 404 returned error can't find the container with id 9b7746c8e2992d6b36dff1265e3fe1a5fc688f7dcc4ce8784805fae49b3fe30f Oct 31 19:36:33.510772 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:33.510748 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b66ef818-fwll4"] Oct 31 19:36:33.762632 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:33.762545 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2242e60d-89ec-4e15-b211-9781a2e1d0a6" path="/var/lib/kubelet/pods/2242e60d-89ec-4e15-b211-9781a2e1d0a6/volumes" Oct 31 19:36:34.295605 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:34.295569 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66ef818-fwll4" event={"ID":"5ad9721c-495b-413e-b60d-29e34471c03d","Type":"ContainerStarted","Data":"c887a357871b0e70a14ed8ea3ffd43e889cd3817395b47c8f4327fa607212849"} Oct 31 19:36:34.296058 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:34.295629 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66ef818-fwll4" event={"ID":"5ad9721c-495b-413e-b60d-29e34471c03d","Type":"ContainerStarted","Data":"9b7746c8e2992d6b36dff1265e3fe1a5fc688f7dcc4ce8784805fae49b3fe30f"} Oct 31 19:36:34.328044 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:34.327991 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66ef818-fwll4" podStartSLOduration=2.327972555 podStartE2EDuration="2.327972555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:34.320884459 +0000 UTC m=+1028.683704832" watchObservedRunningTime="2026-10-31 19:36:34.327972555 +0000 UTC m=+1028.690792930" Oct 31 19:36:35.316324 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:35.316286 4910 generic.go:356] "Generic (PLEG): container finished" podID="5ad9721c-495b-413e-b60d-29e34471c03d" containerID="c887a357871b0e70a14ed8ea3ffd43e889cd3817395b47c8f4327fa607212849" exitCode=0 Oct 31 19:36:35.316745 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:35.316366 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66ef818-fwll4" event={"ID":"5ad9721c-495b-413e-b60d-29e34471c03d","Type":"ContainerDied","Data":"c887a357871b0e70a14ed8ea3ffd43e889cd3817395b47c8f4327fa607212849"} Oct 31 19:36:36.038100 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:36.038011 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7876437-hftq9"] Oct 31 19:36:36.051447 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:36.051405 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7876437-hftq9"] Oct 31 19:36:36.510731 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:36.510706 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66ef818-fwll4" Oct 31 19:36:36.683198 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:36.683105 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ad9721c-495b-413e-b60d-29e34471c03d-kube-api-access-9hmct\" (UniqueName: \"kubernetes.io/projected/5ad9721c-495b-413e-b60d-29e34471c03d-kube-api-access-9hmct\") pod \"5ad9721c-495b-413e-b60d-29e34471c03d\" (UID: \"5ad9721c-495b-413e-b60d-29e34471c03d\") " Oct 31 19:36:36.683480 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:36.683316 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ad9721c-495b-413e-b60d-29e34471c03d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad9721c-495b-413e-b60d-29e34471c03d-maas-api-openshift-ca\") pod \"5ad9721c-495b-413e-b60d-29e34471c03d\" (UID: \"5ad9721c-495b-413e-b60d-29e34471c03d\") " Oct 31 19:36:36.683552 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:36.683538 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ad9721c-495b-413e-b60d-29e34471c03d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad9721c-495b-413e-b60d-29e34471c03d-maas-api-xks-ca\") pod \"5ad9721c-495b-413e-b60d-29e34471c03d\" (UID: \"5ad9721c-495b-413e-b60d-29e34471c03d\") " Oct 31 19:36:36.683665 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:36.683638 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad9721c-495b-413e-b60d-29e34471c03d-maas-api-openshift-ca" pod "5ad9721c-495b-413e-b60d-29e34471c03d" (UID: "5ad9721c-495b-413e-b60d-29e34471c03d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:36:36.684206 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:36.684182 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ad9721c-495b-413e-b60d-29e34471c03d-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:36:36.685367 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:36.685344 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad9721c-495b-413e-b60d-29e34471c03d-kube-api-access-9hmct" pod "5ad9721c-495b-413e-b60d-29e34471c03d" (UID: "5ad9721c-495b-413e-b60d-29e34471c03d"). InnerVolumeSpecName "kube-api-access-9hmct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:36:36.685461 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:36.685445 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ad9721c-495b-413e-b60d-29e34471c03d-maas-api-xks-ca" pod "5ad9721c-495b-413e-b60d-29e34471c03d" (UID: "5ad9721c-495b-413e-b60d-29e34471c03d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:36:36.786201 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:36.786160 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ad9721c-495b-413e-b60d-29e34471c03d-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:36:36.786201 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:36.786201 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hmct\" (UniqueName: \"kubernetes.io/projected/5ad9721c-495b-413e-b60d-29e34471c03d-kube-api-access-9hmct\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:36:37.339485 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:37.339443 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66ef818-fwll4" event={"ID":"5ad9721c-495b-413e-b60d-29e34471c03d","Type":"ContainerDied","Data":"9b7746c8e2992d6b36dff1265e3fe1a5fc688f7dcc4ce8784805fae49b3fe30f"} Oct 31 19:36:37.339485 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:37.339473 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66ef818-fwll4" Oct 31 19:36:37.339701 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:37.339502 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b7746c8e2992d6b36dff1265e3fe1a5fc688f7dcc4ce8784805fae49b3fe30f" Oct 31 19:36:37.756809 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:37.756715 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="100a9e32-a903-4d0d-aed2-e4857230bd7d" path="/var/lib/kubelet/pods/100a9e32-a903-4d0d-aed2-e4857230bd7d/volumes" Oct 31 19:36:42.037454 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:42.037415 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-221d6a18-n42bz"] Oct 31 19:36:42.044788 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:42.044757 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-221d6a18-n42bz"] Oct 31 19:36:42.718291 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:36:42.718244 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:36:43.756810 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:43.756772 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f85466a-6e79-4de7-b5e8-c33d02f708c4" path="/var/lib/kubelet/pods/3f85466a-6e79-4de7-b5e8-c33d02f708c4/volumes" Oct 31 19:36:52.034797 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:52.034764 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06f99b64-h28f7"] Oct 31 19:36:52.039792 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:52.039762 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06f99b64-h28f7"] Oct 31 19:36:53.867999 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:36:53.865877 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2986db8e-72fe-466c-a20a-3ca1a02c0975" path="/var/lib/kubelet/pods/2986db8e-72fe-466c-a20a-3ca1a02c0975/volumes" Oct 31 19:37:00.973054 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:00.973008 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6qm4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:37:00.973718 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:00.973680 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" podUID="c73a891d-f2de-4099-9a98-bdcbae51888f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:37:02.237402 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.237356 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw"] Oct 31 19:37:02.240851 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.238370 4910 cpu_manager.go:414] "containerMap: removing container" podUID="5ad9721c-495b-413e-b60d-29e34471c03d" containerName="revoke-keys" Oct 31 19:37:02.240851 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.238396 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad9721c-495b-413e-b60d-29e34471c03d" containerName="revoke-keys" Oct 31 19:37:02.240851 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.238614 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ad9721c-495b-413e-b60d-29e34471c03d" containerName="revoke-keys" Oct 31 19:37:02.246131 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.244683 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.253261 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.253231 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-67adab47-openshift-default-dockercfg-bbdfj\"" Oct 31 19:37:02.264652 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.264253 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw"] Oct 31 19:37:02.304500 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.304459 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.304702 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.304675 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.304784 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.304723 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.304881 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.304859 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.305019 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.304889 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.305019 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.304953 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.305019 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.305004 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.305200 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.305030 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.305200 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.305078 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bkqs\" (UniqueName: \"kubernetes.io/projected/26ae8974-942b-4bd3-9ca9-f047209ab0fa-kube-api-access-5bkqs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.410162 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.410117 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.410430 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.410410 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.429934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.410607 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.434023 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.433984 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.434282 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.434260 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.434466 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.434450 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.434597 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.434582 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.434759 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.434745 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bkqs\" (UniqueName: \"kubernetes.io/projected/26ae8974-942b-4bd3-9ca9-f047209ab0fa-kube-api-access-5bkqs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.435012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.434993 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.435365 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.426415 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.435508 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.435480 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.435676 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.426729 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.502069 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.445598 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.502303 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.434759 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.548021 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.547951 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bkqs\" (UniqueName: \"kubernetes.io/projected/26ae8974-942b-4bd3-9ca9-f047209ab0fa-kube-api-access-5bkqs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.548515 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.548468 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.549189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.549145 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.608189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.608091 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:02.903901 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:02.903861 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:03.277166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:03.277119 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:37:03.277572 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:03.277217 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:37:03.277572 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:03.277259 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:37:03.277848 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:03.277824 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw"] Oct 31 19:37:03.874283 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:03.874242 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" event={"ID":"26ae8974-942b-4bd3-9ca9-f047209ab0fa","Type":"ContainerStarted","Data":"e5e35193844c1dd27ba0c3731bd00c02d0979f62c443b1682e9e77ea3fe97e88"} Oct 31 19:37:03.874504 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:03.874295 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" event={"ID":"26ae8974-942b-4bd3-9ca9-f047209ab0fa","Type":"ContainerStarted","Data":"ddbf29cbfca49d8f83560ff58cb07b5d99ebe0a3477a5769ce033cceb6c443b1"} Oct 31 19:37:03.903982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:03.903949 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:03.908633 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:03.908581 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" podStartSLOduration=1.908567024 podStartE2EDuration="1.908567024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:37:03.904020004 +0000 UTC m=+1058.266840375" watchObservedRunningTime="2026-10-31 19:37:03.908567024 +0000 UTC m=+1058.271387396" Oct 31 19:37:04.077107 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:04.077065 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7ca570a-z76dr"] Oct 31 19:37:04.086307 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:04.084120 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7ca570a-z76dr"] Oct 31 19:37:04.910307 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:04.910277 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:05.783978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:05.780721 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="467f91c0-a951-4529-83be-055231ca73a4" path="/var/lib/kubelet/pods/467f91c0-a951-4529-83be-055231ca73a4/volumes" Oct 31 19:37:05.893782 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:05.893745 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:05.894845 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:05.894822 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:37:12.715671 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:37:12.715633 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:37:23.699142 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:23.699061 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeb94931-j9mpm"] Oct 31 19:37:23.703719 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:23.703699 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb94931-j9mpm" Oct 31 19:37:23.713243 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:23.713217 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeb94931-j9mpm"] Oct 31 19:37:23.742110 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:23.742076 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:37:23.742290 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:23.742262 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:37:23.803222 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:23.803184 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpg2s\" (UniqueName: \"kubernetes.io/projected/38b16441-a7cb-4418-9306-13d4c448f7d4-kube-api-access-cpg2s\") pod \"maas-api-revoke-sub-aeb94931-j9mpm\" (UID: \"38b16441-a7cb-4418-9306-13d4c448f7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb94931-j9mpm" Oct 31 19:37:23.803393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:23.803310 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b16441-a7cb-4418-9306-13d4c448f7d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeb94931-j9mpm\" (UID: \"38b16441-a7cb-4418-9306-13d4c448f7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb94931-j9mpm" Oct 31 19:37:23.803393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:23.803341 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b16441-a7cb-4418-9306-13d4c448f7d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeb94931-j9mpm\" (UID: \"38b16441-a7cb-4418-9306-13d4c448f7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb94931-j9mpm" Oct 31 19:37:23.908207 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:23.905941 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpg2s\" (UniqueName: \"kubernetes.io/projected/38b16441-a7cb-4418-9306-13d4c448f7d4-kube-api-access-cpg2s\") pod \"maas-api-revoke-sub-aeb94931-j9mpm\" (UID: \"38b16441-a7cb-4418-9306-13d4c448f7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb94931-j9mpm" Oct 31 19:37:23.908207 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:23.906115 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b16441-a7cb-4418-9306-13d4c448f7d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeb94931-j9mpm\" (UID: \"38b16441-a7cb-4418-9306-13d4c448f7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb94931-j9mpm" Oct 31 19:37:23.908207 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:23.906144 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b16441-a7cb-4418-9306-13d4c448f7d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeb94931-j9mpm\" (UID: \"38b16441-a7cb-4418-9306-13d4c448f7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb94931-j9mpm" Oct 31 19:37:23.908749 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:23.908719 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b16441-a7cb-4418-9306-13d4c448f7d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeb94931-j9mpm\" (UID: \"38b16441-a7cb-4418-9306-13d4c448f7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb94931-j9mpm" Oct 31 19:37:23.926808 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:23.926718 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b16441-a7cb-4418-9306-13d4c448f7d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeb94931-j9mpm\" (UID: \"38b16441-a7cb-4418-9306-13d4c448f7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb94931-j9mpm" Oct 31 19:37:23.932737 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:23.932705 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpg2s\" (UniqueName: \"kubernetes.io/projected/38b16441-a7cb-4418-9306-13d4c448f7d4-kube-api-access-cpg2s\") pod \"maas-api-revoke-sub-aeb94931-j9mpm\" (UID: \"38b16441-a7cb-4418-9306-13d4c448f7d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb94931-j9mpm" Oct 31 19:37:24.045054 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:24.044977 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb94931-j9mpm" Oct 31 19:37:24.332891 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:37:24.332321 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b16441_a7cb_4418_9306_13d4c448f7d4.slice/crio-6be4ba9b851116766407a6b242c019b0e48c19b2895cd172f98e26d6306a9165 WatchSource:0}: Error finding container 6be4ba9b851116766407a6b242c019b0e48c19b2895cd172f98e26d6306a9165: Status 404 returned error can't find the container with id 6be4ba9b851116766407a6b242c019b0e48c19b2895cd172f98e26d6306a9165 Oct 31 19:37:24.335354 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:24.335326 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeb94931-j9mpm"] Oct 31 19:37:25.359642 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:25.359610 4910 generic.go:356] "Generic (PLEG): container finished" podID="38b16441-a7cb-4418-9306-13d4c448f7d4" containerID="bd9b3693136852d4068e6cc9c75481166f672c8cc6b83e8a2e68b19d431c5e05" exitCode=0 Oct 31 19:37:25.360082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:25.359697 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb94931-j9mpm" event={"ID":"38b16441-a7cb-4418-9306-13d4c448f7d4","Type":"ContainerDied","Data":"bd9b3693136852d4068e6cc9c75481166f672c8cc6b83e8a2e68b19d431c5e05"} Oct 31 19:37:25.360082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:25.359752 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb94931-j9mpm" event={"ID":"38b16441-a7cb-4418-9306-13d4c448f7d4","Type":"ContainerStarted","Data":"6be4ba9b851116766407a6b242c019b0e48c19b2895cd172f98e26d6306a9165"} Oct 31 19:37:26.591271 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:26.591206 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb94931-j9mpm" Oct 31 19:37:26.673390 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:26.666205 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38b16441-a7cb-4418-9306-13d4c448f7d4-kube-api-access-cpg2s\" (UniqueName: \"kubernetes.io/projected/38b16441-a7cb-4418-9306-13d4c448f7d4-kube-api-access-cpg2s\") pod \"38b16441-a7cb-4418-9306-13d4c448f7d4\" (UID: \"38b16441-a7cb-4418-9306-13d4c448f7d4\") " Oct 31 19:37:26.673390 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:26.666344 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38b16441-a7cb-4418-9306-13d4c448f7d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b16441-a7cb-4418-9306-13d4c448f7d4-maas-api-openshift-ca\") pod \"38b16441-a7cb-4418-9306-13d4c448f7d4\" (UID: \"38b16441-a7cb-4418-9306-13d4c448f7d4\") " Oct 31 19:37:26.673390 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:26.666476 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38b16441-a7cb-4418-9306-13d4c448f7d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b16441-a7cb-4418-9306-13d4c448f7d4-maas-api-xks-ca\") pod \"38b16441-a7cb-4418-9306-13d4c448f7d4\" (UID: \"38b16441-a7cb-4418-9306-13d4c448f7d4\") " Oct 31 19:37:26.673390 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:26.667940 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38b16441-a7cb-4418-9306-13d4c448f7d4-maas-api-openshift-ca" pod "38b16441-a7cb-4418-9306-13d4c448f7d4" (UID: "38b16441-a7cb-4418-9306-13d4c448f7d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:37:26.673390 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:26.669277 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38b16441-a7cb-4418-9306-13d4c448f7d4-kube-api-access-cpg2s" pod "38b16441-a7cb-4418-9306-13d4c448f7d4" (UID: "38b16441-a7cb-4418-9306-13d4c448f7d4"). InnerVolumeSpecName "kube-api-access-cpg2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:37:26.673390 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:26.670947 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38b16441-a7cb-4418-9306-13d4c448f7d4-maas-api-xks-ca" pod "38b16441-a7cb-4418-9306-13d4c448f7d4" (UID: "38b16441-a7cb-4418-9306-13d4c448f7d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:37:26.768278 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:26.768238 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b16441-a7cb-4418-9306-13d4c448f7d4-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:37:26.768278 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:26.768274 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpg2s\" (UniqueName: \"kubernetes.io/projected/38b16441-a7cb-4418-9306-13d4c448f7d4-kube-api-access-cpg2s\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:37:26.768278 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:26.768283 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b16441-a7cb-4418-9306-13d4c448f7d4-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:37:27.111384 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:27.111347 4910 scope.go:122] "RemoveContainer" containerID="cd87e8b20ba7e71c08fe6b9ee453ead947cb6423ecebcff5b07827b03566a537" Oct 31 19:37:27.126534 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:27.126511 4910 scope.go:122] "RemoveContainer" containerID="dae0a9ccec4f71bc359c483b8d6055f4de77c501a79f85b9620aadf258bb7ab5" Oct 31 19:37:27.137095 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:27.137071 4910 scope.go:122] "RemoveContainer" containerID="9c1d9375aed24c7ffe1937099069e0b7dd131de486e268f53099d456a7f3ccef" Oct 31 19:37:27.148845 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:27.148821 4910 scope.go:122] "RemoveContainer" containerID="4f759eb2ccece62f8ce8084cac067123e3a2a63fe10fa840a5695ec89b3190da" Oct 31 19:37:27.160544 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:27.160524 4910 scope.go:122] "RemoveContainer" containerID="c197d11e536b593a302211b9fb17668a3e800375744fad71ca672c6eac8085f2" Oct 31 19:37:27.171043 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:27.171022 4910 scope.go:122] "RemoveContainer" containerID="22121b1178f4795221390bf79204a19b9d9cde91df292af141720ffb4776815b" Oct 31 19:37:27.181649 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:27.181630 4910 scope.go:122] "RemoveContainer" containerID="3e2bea4e07f6b0b6a43f4e3dc0182d3d5723eb214ddd004fe3d5f8d2eafe0900" Oct 31 19:37:27.191740 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:27.191717 4910 scope.go:122] "RemoveContainer" containerID="acb48e1267e383e7679757fd7d0c3777809c5a9cb91ceaf39560016edd44dd42" Oct 31 19:37:27.202240 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:27.202218 4910 scope.go:122] "RemoveContainer" containerID="4316831fea2a129e74ad1748aecb5388ccd9d60b314f9654c41aa4292e25419a" Oct 31 19:37:27.212303 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:27.212280 4910 scope.go:122] "RemoveContainer" containerID="c65c970bd8053e84f6c5dafc387a56afea05c274667d6a00f48183e60c228558" Oct 31 19:37:27.382810 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:27.382714 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb94931-j9mpm" event={"ID":"38b16441-a7cb-4418-9306-13d4c448f7d4","Type":"ContainerDied","Data":"6be4ba9b851116766407a6b242c019b0e48c19b2895cd172f98e26d6306a9165"} Oct 31 19:37:27.382810 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:27.382773 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6be4ba9b851116766407a6b242c019b0e48c19b2895cd172f98e26d6306a9165" Oct 31 19:37:27.382810 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:27.382726 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb94931-j9mpm" Oct 31 19:37:42.713863 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:37:42.713825 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:37:49.038294 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:49.038257 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f75ea166-4h57n"] Oct 31 19:37:49.044866 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:49.044829 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f75ea166-4h57n"] Oct 31 19:37:49.758165 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:49.758120 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc1c443a-03de-4a57-ab05-265775a8ba1a" path="/var/lib/kubelet/pods/cc1c443a-03de-4a57-ab05-265775a8ba1a/volumes" Oct 31 19:37:55.031839 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:55.031798 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6ca1252-5nwbd"] Oct 31 19:37:55.036816 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:55.036764 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6ca1252-5nwbd"] Oct 31 19:37:55.761502 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:37:55.761458 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2056a6d-547a-4aeb-b054-ca3ceb602d4f" path="/var/lib/kubelet/pods/a2056a6d-547a-4aeb-b054-ca3ceb602d4f/volumes" Oct 31 19:38:03.034434 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:03.034385 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9491c70-m9xkz"] Oct 31 19:38:03.039760 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:03.039730 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9491c70-m9xkz"] Oct 31 19:38:03.757938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:03.757879 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0" path="/var/lib/kubelet/pods/ed2cdc6d-4017-4a4b-baf3-7726ba94e1b0/volumes" Oct 31 19:38:08.091478 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:08.091358 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-150e73ff-62hcr"] Oct 31 19:38:08.158464 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:08.158421 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-150e73ff-62hcr"] Oct 31 19:38:08.486947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:08.486077 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:08.542341 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:08.539458 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:08.691936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:08.681898 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:08.728047 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:08.727995 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:08.728792 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:08.728755 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:08.919805 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:08.919761 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:38:08.936039 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:08.935992 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:08.975079 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:08.975006 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:09.104050 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:09.085183 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:09.112932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:09.105030 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:09.123944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:09.115335 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:09.124679 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:09.124624 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:38:09.125211 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:09.125185 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:38:09.497087 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:09.497028 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:09.497824 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:09.497796 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:38:09.656007 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:09.647157 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:09.708384 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:09.683049 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:09.732935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:09.709019 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:38:09.818932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:09.731131 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:09.818932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:09.782044 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:38:09.924544 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:09.924495 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:09.942380 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:09.941346 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:09.978377 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:09.977314 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:10.090987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:10.088007 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:10.093937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:10.091937 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a49b819-0081-411c-87d4-8a703c754708" path="/var/lib/kubelet/pods/8a49b819-0081-411c-87d4-8a703c754708/volumes" Oct 31 19:38:10.093937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:10.092720 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:38:10.093937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:10.092829 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:38:10.093937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:10.092866 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:38:10.093937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:10.092903 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:38:10.093937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:10.093009 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:38:10.093937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:10.093043 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:38:10.114429 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:10.112575 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:38:10.146372 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:10.143508 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:10.146372 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:10.143821 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:38:10.191939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:10.186105 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:10.191939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:10.186570 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:38:10.216157 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:10.213688 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:38:10.237639 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:10.236055 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:38:10.237639 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:10.236121 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:38:10.270005 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:10.269895 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:38:10.295494 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:10.295044 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:38:10.297812 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:10.297728 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:38:10.319054 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:10.316764 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:38:12.721485 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:38:12.721448 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:38:14.140408 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.140375 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9"] Oct 31 19:38:14.141974 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.141944 4910 cpu_manager.go:414] "containerMap: removing container" podUID="38b16441-a7cb-4418-9306-13d4c448f7d4" containerName="revoke-keys" Oct 31 19:38:14.142127 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.142114 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38b16441-a7cb-4418-9306-13d4c448f7d4" containerName="revoke-keys" Oct 31 19:38:14.142692 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.142407 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="38b16441-a7cb-4418-9306-13d4c448f7d4" containerName="revoke-keys" Oct 31 19:38:14.154152 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.154125 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.159142 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.159095 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-db61066b-openshift-default-dockercfg-8msmt\"" Oct 31 19:38:14.175608 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.175575 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9"] Oct 31 19:38:14.380231 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.375518 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxt2p\" (UniqueName: \"kubernetes.io/projected/790bd78d-4525-4587-9b6f-30af5841a88a-kube-api-access-vxt2p\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.380231 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.375713 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790bd78d-4525-4587-9b6f-30af5841a88a-istio-podinfo\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.380231 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.375756 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-istio-envoy\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.380231 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.375803 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790bd78d-4525-4587-9b6f-30af5841a88a-istiod-ca-cert\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.380231 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.375904 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-workload-certs\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.380231 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.375954 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-workload-socket\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.380231 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.376021 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790bd78d-4525-4587-9b6f-30af5841a88a-istio-token\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.380231 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.376061 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-istio-data\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.380231 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.376167 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-credential-socket\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.494987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.494075 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-credential-socket\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.494987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.494236 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxt2p\" (UniqueName: \"kubernetes.io/projected/790bd78d-4525-4587-9b6f-30af5841a88a-kube-api-access-vxt2p\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.494987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.494336 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790bd78d-4525-4587-9b6f-30af5841a88a-istio-podinfo\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.494987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.494375 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-istio-envoy\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.494987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.494414 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790bd78d-4525-4587-9b6f-30af5841a88a-istiod-ca-cert\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.494987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.494506 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-workload-certs\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.494987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.494538 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-workload-socket\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.494987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.494586 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790bd78d-4525-4587-9b6f-30af5841a88a-istio-token\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.494987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.494624 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-istio-data\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.495509 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.495108 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-istio-data\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.495509 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.495390 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-credential-socket\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.502637 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.501043 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790bd78d-4525-4587-9b6f-30af5841a88a-istiod-ca-cert\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.525516 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.525476 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-workload-certs\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.539530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.538328 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-workload-socket\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.601783 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.601681 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790bd78d-4525-4587-9b6f-30af5841a88a-istio-podinfo\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.608572 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.606748 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790bd78d-4525-4587-9b6f-30af5841a88a-istio-token\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.608572 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.607023 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-istio-envoy\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.611979 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.611884 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxt2p\" (UniqueName: \"kubernetes.io/projected/790bd78d-4525-4587-9b6f-30af5841a88a-kube-api-access-vxt2p\") pod \"e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:14.780958 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:14.780828 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:15.118626 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:38:15.118193 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790bd78d_4525_4587_9b6f_30af5841a88a.slice/crio-9b3451259462fea753f446071eef6fa840e0f2832d86fd2f646d8944ce110df4 WatchSource:0}: Error finding container 9b3451259462fea753f446071eef6fa840e0f2832d86fd2f646d8944ce110df4: Status 404 returned error can't find the container with id 9b3451259462fea753f446071eef6fa840e0f2832d86fd2f646d8944ce110df4 Oct 31 19:38:15.118626 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:15.118435 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9"] Oct 31 19:38:15.121953 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:15.121896 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:38:15.122065 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:15.122002 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:38:15.122130 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:15.122068 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:38:15.554407 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:15.554322 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" event={"ID":"790bd78d-4525-4587-9b6f-30af5841a88a","Type":"ContainerStarted","Data":"db127c0d1f3a7531ff62c648fe12abfcc64371e36d96e7f50f8bcfeadc1ff6a7"} Oct 31 19:38:15.555019 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:15.554941 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" event={"ID":"790bd78d-4525-4587-9b6f-30af5841a88a","Type":"ContainerStarted","Data":"9b3451259462fea753f446071eef6fa840e0f2832d86fd2f646d8944ce110df4"} Oct 31 19:38:15.585626 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:15.584211 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" podStartSLOduration=1.584196564 podStartE2EDuration="1.584196564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:15.581395775 +0000 UTC m=+1129.944216145" watchObservedRunningTime="2026-10-31 19:38:15.584196564 +0000 UTC m=+1129.947016947" Oct 31 19:38:15.787022 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:15.785674 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:16.567836 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:16.567802 4910 generic.go:356] "Generic (PLEG): container finished" podID="26ae8974-942b-4bd3-9ca9-f047209ab0fa" containerID="e5e35193844c1dd27ba0c3731bd00c02d0979f62c443b1682e9e77ea3fe97e88" exitCode=0 Oct 31 19:38:16.568408 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:16.567880 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" event={"ID":"26ae8974-942b-4bd3-9ca9-f047209ab0fa","Type":"ContainerDied","Data":"e5e35193844c1dd27ba0c3731bd00c02d0979f62c443b1682e9e77ea3fe97e88"} Oct 31 19:38:16.568995 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:16.568971 4910 scope.go:122] "RemoveContainer" containerID="e5e35193844c1dd27ba0c3731bd00c02d0979f62c443b1682e9e77ea3fe97e88" Oct 31 19:38:16.570890 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:16.570861 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:38:16.571021 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:16.570946 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:38:16.571021 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:16.571013 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:38:16.788619 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:16.788535 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:17.588103 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:17.588047 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" event={"ID":"26ae8974-942b-4bd3-9ca9-f047209ab0fa","Type":"ContainerStarted","Data":"d5ca693c42416ee872e735614fbde751da9997d42c679a8f1207cce9f047313e"} Oct 31 19:38:17.589140 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:17.588127 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:17.601971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:17.598283 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:38:17.904090 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:17.904013 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:38:17.905146 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:17.905119 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 19:38:17.905283 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:17.905173 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" podUID="26ae8974-942b-4bd3-9ca9-f047209ab0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 19:38:18.904881 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:18.904840 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 19:38:18.905358 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:18.904905 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" podUID="26ae8974-942b-4bd3-9ca9-f047209ab0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 19:38:19.916933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:19.908090 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 19:38:19.916933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:19.908161 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" podUID="26ae8974-942b-4bd3-9ca9-f047209ab0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 19:38:20.894462 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:20.894426 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:38:20.904306 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:20.904272 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 19:38:20.904459 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:20.904325 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" podUID="26ae8974-942b-4bd3-9ca9-f047209ab0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 19:38:21.187943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.187831 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc"] Oct 31 19:38:21.195248 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.195003 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.196172 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.196147 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc"] Oct 31 19:38:21.206955 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.204697 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-4bac5a-openshift-default-dockercfg-p2t4w\"" Oct 31 19:38:21.318937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.310069 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922b063f-4aee-4ed2-9355-3cd5222862df-istiod-ca-cert\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.318937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.310127 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-workload-socket\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.318937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.310169 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-istio-envoy\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.318937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.310270 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-istio-data\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.318937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.310382 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97z6l\" (UniqueName: \"kubernetes.io/projected/922b063f-4aee-4ed2-9355-3cd5222862df-kube-api-access-97z6l\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.318937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.310421 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922b063f-4aee-4ed2-9355-3cd5222862df-istio-token\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.318937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.310467 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-credential-socket\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.318937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.310541 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922b063f-4aee-4ed2-9355-3cd5222862df-istio-podinfo\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.318937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.310673 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-workload-certs\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.493523 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.475201 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-workload-certs\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.493523 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.475293 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922b063f-4aee-4ed2-9355-3cd5222862df-istiod-ca-cert\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.493523 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.475328 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-workload-socket\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.493523 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.475364 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-istio-envoy\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.493523 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.475502 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-istio-data\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.493523 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.475657 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97z6l\" (UniqueName: \"kubernetes.io/projected/922b063f-4aee-4ed2-9355-3cd5222862df-kube-api-access-97z6l\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.493523 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.475723 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-workload-certs\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.493523 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.475996 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-workload-socket\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.493523 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.476636 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922b063f-4aee-4ed2-9355-3cd5222862df-istiod-ca-cert\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.493523 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.477199 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-istio-data\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.493523 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.477266 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922b063f-4aee-4ed2-9355-3cd5222862df-istio-token\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.493523 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.477323 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-credential-socket\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.493523 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.477411 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922b063f-4aee-4ed2-9355-3cd5222862df-istio-podinfo\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.541965 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.505108 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-credential-socket\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.556935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.548623 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97z6l\" (UniqueName: \"kubernetes.io/projected/922b063f-4aee-4ed2-9355-3cd5222862df-kube-api-access-97z6l\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.574931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.571465 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922b063f-4aee-4ed2-9355-3cd5222862df-istio-token\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.584936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.584094 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-istio-envoy\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.609063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.600579 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922b063f-4aee-4ed2-9355-3cd5222862df-istio-podinfo\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.874931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.873415 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:21.916291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.913961 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 19:38:21.916291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:21.914030 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" podUID="26ae8974-942b-4bd3-9ca9-f047209ab0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 19:38:22.311667 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:38:22.311612 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922b063f_4aee_4ed2_9355_3cd5222862df.slice/crio-f5eeb8b7349c1f80ec72108d602c24485c3d341a05a718a54733c7867f911ad8 WatchSource:0}: Error finding container f5eeb8b7349c1f80ec72108d602c24485c3d341a05a718a54733c7867f911ad8: Status 404 returned error can't find the container with id f5eeb8b7349c1f80ec72108d602c24485c3d341a05a718a54733c7867f911ad8 Oct 31 19:38:22.312327 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:22.312216 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc"] Oct 31 19:38:22.315462 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:22.315415 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:38:22.315568 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:22.315543 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:38:22.315620 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:22.315605 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:38:22.924865 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:22.924825 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 19:38:22.925176 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:22.925131 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" podUID="26ae8974-942b-4bd3-9ca9-f047209ab0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 19:38:22.992966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:22.990611 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" event={"ID":"922b063f-4aee-4ed2-9355-3cd5222862df","Type":"ContainerStarted","Data":"f40141a0f0edd3d9dbf89b3b2852783ef462ab01f27fe56d3f41f5db6e47b93a"} Oct 31 19:38:22.992966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:22.990715 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" event={"ID":"922b063f-4aee-4ed2-9355-3cd5222862df","Type":"ContainerStarted","Data":"f5eeb8b7349c1f80ec72108d602c24485c3d341a05a718a54733c7867f911ad8"} Oct 31 19:38:23.070370 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:23.048880 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" podStartSLOduration=2.048859349 podStartE2EDuration="2.048859349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:23.030268571 +0000 UTC m=+1137.393088944" watchObservedRunningTime="2026-10-31 19:38:23.048859349 +0000 UTC m=+1137.411679724" Oct 31 19:38:23.877800 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:23.877756 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:23.884582 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:23.884548 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:23.904644 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:23.904612 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 19:38:23.904860 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:23.904660 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" podUID="26ae8974-942b-4bd3-9ca9-f047209ab0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 19:38:24.008618 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:24.005624 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:24.008618 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:24.007564 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:38:24.961935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:24.960903 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:38:24.965935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:24.964636 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:38:25.501878 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:25.501841 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1568f942-s7wjr"] Oct 31 19:38:25.506804 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:25.506781 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1568f942-s7wjr" Oct 31 19:38:25.514398 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:25.513990 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:38:25.514398 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:25.514029 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:38:25.535059 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:25.528277 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5sk4\" (UniqueName: \"kubernetes.io/projected/3ab18638-5a80-47e6-9ef0-02031de59836-kube-api-access-s5sk4\") pod \"maas-api-revoke-sub-1568f942-s7wjr\" (UID: \"3ab18638-5a80-47e6-9ef0-02031de59836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1568f942-s7wjr" Oct 31 19:38:25.535059 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:25.528527 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab18638-5a80-47e6-9ef0-02031de59836-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1568f942-s7wjr\" (UID: \"3ab18638-5a80-47e6-9ef0-02031de59836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1568f942-s7wjr" Oct 31 19:38:25.535059 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:25.528590 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab18638-5a80-47e6-9ef0-02031de59836-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1568f942-s7wjr\" (UID: \"3ab18638-5a80-47e6-9ef0-02031de59836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1568f942-s7wjr" Oct 31 19:38:25.579945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:25.579842 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1568f942-s7wjr"] Oct 31 19:38:25.631253 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:25.630834 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab18638-5a80-47e6-9ef0-02031de59836-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1568f942-s7wjr\" (UID: \"3ab18638-5a80-47e6-9ef0-02031de59836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1568f942-s7wjr" Oct 31 19:38:25.631253 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:25.630959 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab18638-5a80-47e6-9ef0-02031de59836-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1568f942-s7wjr\" (UID: \"3ab18638-5a80-47e6-9ef0-02031de59836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1568f942-s7wjr" Oct 31 19:38:25.631710 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:25.631572 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5sk4\" (UniqueName: \"kubernetes.io/projected/3ab18638-5a80-47e6-9ef0-02031de59836-kube-api-access-s5sk4\") pod \"maas-api-revoke-sub-1568f942-s7wjr\" (UID: \"3ab18638-5a80-47e6-9ef0-02031de59836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1568f942-s7wjr" Oct 31 19:38:25.631824 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:25.631779 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab18638-5a80-47e6-9ef0-02031de59836-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1568f942-s7wjr\" (UID: \"3ab18638-5a80-47e6-9ef0-02031de59836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1568f942-s7wjr" Oct 31 19:38:25.641422 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:25.641388 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab18638-5a80-47e6-9ef0-02031de59836-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1568f942-s7wjr\" (UID: \"3ab18638-5a80-47e6-9ef0-02031de59836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1568f942-s7wjr" Oct 31 19:38:25.659250 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:25.659113 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5sk4\" (UniqueName: \"kubernetes.io/projected/3ab18638-5a80-47e6-9ef0-02031de59836-kube-api-access-s5sk4\") pod \"maas-api-revoke-sub-1568f942-s7wjr\" (UID: \"3ab18638-5a80-47e6-9ef0-02031de59836\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1568f942-s7wjr" Oct 31 19:38:25.855321 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:25.855286 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:38:25.860291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:25.860245 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1568f942-s7wjr" Oct 31 19:38:26.387745 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:38:26.387713 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab18638_5a80_47e6_9ef0_02031de59836.slice/crio-c8fb20d141d15c1da8a0989820d7d71138af4a5e8293e9acc62df5c7afadf024 WatchSource:0}: Error finding container c8fb20d141d15c1da8a0989820d7d71138af4a5e8293e9acc62df5c7afadf024: Status 404 returned error can't find the container with id c8fb20d141d15c1da8a0989820d7d71138af4a5e8293e9acc62df5c7afadf024 Oct 31 19:38:26.389779 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:26.389755 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1568f942-s7wjr"] Oct 31 19:38:27.093966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:27.093863 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1568f942-s7wjr" event={"ID":"3ab18638-5a80-47e6-9ef0-02031de59836","Type":"ContainerStarted","Data":"90a8064849484831733d01b8c0bbdbd104a593b72722b4c86c1ab824d6b6c185"} Oct 31 19:38:27.093966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:27.093937 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1568f942-s7wjr" event={"ID":"3ab18638-5a80-47e6-9ef0-02031de59836","Type":"ContainerStarted","Data":"c8fb20d141d15c1da8a0989820d7d71138af4a5e8293e9acc62df5c7afadf024"} Oct 31 19:38:27.155312 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:27.155242 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1568f942-s7wjr" podStartSLOduration=2.155218783 podStartE2EDuration="2.155218783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:27.139985437 +0000 UTC m=+1141.502805809" watchObservedRunningTime="2026-10-31 19:38:27.155218783 +0000 UTC m=+1141.518039156" Oct 31 19:38:27.355626 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:27.355543 4910 scope.go:122] "RemoveContainer" containerID="13e7d2335909412834515f84beb0496854bbfbddd530460a72b536006f6ccab6" Oct 31 19:38:27.374843 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:27.374815 4910 scope.go:122] "RemoveContainer" containerID="ff9736032812c377a7579c6852aed0b9aa2bc88556c2501841c7516e8a85f0de" Oct 31 19:38:27.386762 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:27.386734 4910 scope.go:122] "RemoveContainer" containerID="1f95e9e0cc67b0d5acaa5ad4fa07ca0f6fb00e3a121512297ebd19eb8b273f1d" Oct 31 19:38:27.397569 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:27.397462 4910 scope.go:122] "RemoveContainer" containerID="36f728cceae288865378ac145363f35797c62dbdbae13d8fad74c8460c1abe97" Oct 31 19:38:27.410494 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:27.410463 4910 scope.go:122] "RemoveContainer" containerID="400272b45d4e6674d8b50b140705ecfd8ea0ca6d717017884b37985896716b72" Oct 31 19:38:28.104541 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:28.104503 4910 generic.go:356] "Generic (PLEG): container finished" podID="3ab18638-5a80-47e6-9ef0-02031de59836" containerID="90a8064849484831733d01b8c0bbdbd104a593b72722b4c86c1ab824d6b6c185" exitCode=0 Oct 31 19:38:28.104739 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:28.104592 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1568f942-s7wjr" event={"ID":"3ab18638-5a80-47e6-9ef0-02031de59836","Type":"ContainerDied","Data":"90a8064849484831733d01b8c0bbdbd104a593b72722b4c86c1ab824d6b6c185"} Oct 31 19:38:29.038008 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:29.037972 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60e0aed9-5j9f4"] Oct 31 19:38:29.047181 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:29.047150 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60e0aed9-5j9f4"] Oct 31 19:38:29.333421 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:29.333386 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1568f942-s7wjr" Oct 31 19:38:29.450288 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:29.450254 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ab18638-5a80-47e6-9ef0-02031de59836-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab18638-5a80-47e6-9ef0-02031de59836-maas-api-xks-ca\") pod \"3ab18638-5a80-47e6-9ef0-02031de59836\" (UID: \"3ab18638-5a80-47e6-9ef0-02031de59836\") " Oct 31 19:38:29.450454 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:29.450419 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ab18638-5a80-47e6-9ef0-02031de59836-kube-api-access-s5sk4\" (UniqueName: \"kubernetes.io/projected/3ab18638-5a80-47e6-9ef0-02031de59836-kube-api-access-s5sk4\") pod \"3ab18638-5a80-47e6-9ef0-02031de59836\" (UID: \"3ab18638-5a80-47e6-9ef0-02031de59836\") " Oct 31 19:38:29.450522 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:29.450506 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ab18638-5a80-47e6-9ef0-02031de59836-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab18638-5a80-47e6-9ef0-02031de59836-maas-api-openshift-ca\") pod \"3ab18638-5a80-47e6-9ef0-02031de59836\" (UID: \"3ab18638-5a80-47e6-9ef0-02031de59836\") " Oct 31 19:38:29.450896 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:29.450863 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ab18638-5a80-47e6-9ef0-02031de59836-maas-api-openshift-ca" pod "3ab18638-5a80-47e6-9ef0-02031de59836" (UID: "3ab18638-5a80-47e6-9ef0-02031de59836"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:38:29.451419 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:29.451393 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab18638-5a80-47e6-9ef0-02031de59836-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:38:29.452515 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:29.452484 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ab18638-5a80-47e6-9ef0-02031de59836-maas-api-xks-ca" pod "3ab18638-5a80-47e6-9ef0-02031de59836" (UID: "3ab18638-5a80-47e6-9ef0-02031de59836"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:38:29.452644 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:29.452624 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ab18638-5a80-47e6-9ef0-02031de59836-kube-api-access-s5sk4" pod "3ab18638-5a80-47e6-9ef0-02031de59836" (UID: "3ab18638-5a80-47e6-9ef0-02031de59836"). InnerVolumeSpecName "kube-api-access-s5sk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:38:29.552845 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:29.552746 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab18638-5a80-47e6-9ef0-02031de59836-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:38:29.552845 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:29.552778 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5sk4\" (UniqueName: \"kubernetes.io/projected/3ab18638-5a80-47e6-9ef0-02031de59836-kube-api-access-s5sk4\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:38:29.757191 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:29.757157 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eed7b17-62bf-41b8-90bf-95c5ec361bc3" path="/var/lib/kubelet/pods/8eed7b17-62bf-41b8-90bf-95c5ec361bc3/volumes" Oct 31 19:38:30.137667 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:30.137637 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1568f942-s7wjr" Oct 31 19:38:30.138053 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:30.137674 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1568f942-s7wjr" event={"ID":"3ab18638-5a80-47e6-9ef0-02031de59836","Type":"ContainerDied","Data":"c8fb20d141d15c1da8a0989820d7d71138af4a5e8293e9acc62df5c7afadf024"} Oct 31 19:38:30.138053 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:30.137721 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8fb20d141d15c1da8a0989820d7d71138af4a5e8293e9acc62df5c7afadf024" Oct 31 19:38:32.219254 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:32.219213 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98"] Oct 31 19:38:32.221470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:32.221439 4910 cpu_manager.go:414] "containerMap: removing container" podUID="3ab18638-5a80-47e6-9ef0-02031de59836" containerName="revoke-keys" Oct 31 19:38:32.221657 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:32.221642 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ab18638-5a80-47e6-9ef0-02031de59836" containerName="revoke-keys" Oct 31 19:38:32.222209 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:32.222188 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ab18638-5a80-47e6-9ef0-02031de59836" containerName="revoke-keys" Oct 31 19:38:32.251404 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:32.251369 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98"] Oct 31 19:38:32.251667 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:32.251531 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98" Oct 31 19:38:32.257019 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:32.256763 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-db61066b-serving-cert\"" Oct 31 19:38:32.299145 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:32.299105 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz6ht\" (UniqueName: \"kubernetes.io/projected/505b718c-f2ab-4f6d-892a-ed614bc2335d-kube-api-access-lz6ht\") pod \"maas-api-e2e-mt-db61066b-58d9b4d459-zlw98\" (UID: \"505b718c-f2ab-4f6d-892a-ed614bc2335d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98" Oct 31 19:38:32.299437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:32.299288 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/505b718c-f2ab-4f6d-892a-ed614bc2335d-openshift-service-ca\") pod \"maas-api-e2e-mt-db61066b-58d9b4d459-zlw98\" (UID: \"505b718c-f2ab-4f6d-892a-ed614bc2335d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98" Oct 31 19:38:32.299437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:32.299340 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/505b718c-f2ab-4f6d-892a-ed614bc2335d-maas-api-tls\") pod \"maas-api-e2e-mt-db61066b-58d9b4d459-zlw98\" (UID: \"505b718c-f2ab-4f6d-892a-ed614bc2335d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98" Oct 31 19:38:32.299437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:32.299407 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/505b718c-f2ab-4f6d-892a-ed614bc2335d-metrics-certs\") pod \"maas-api-e2e-mt-db61066b-58d9b4d459-zlw98\" (UID: \"505b718c-f2ab-4f6d-892a-ed614bc2335d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98" Oct 31 19:38:32.425265 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:32.420980 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/505b718c-f2ab-4f6d-892a-ed614bc2335d-metrics-certs\") pod \"maas-api-e2e-mt-db61066b-58d9b4d459-zlw98\" (UID: \"505b718c-f2ab-4f6d-892a-ed614bc2335d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98" Oct 31 19:38:32.425803 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:32.425766 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz6ht\" (UniqueName: \"kubernetes.io/projected/505b718c-f2ab-4f6d-892a-ed614bc2335d-kube-api-access-lz6ht\") pod \"maas-api-e2e-mt-db61066b-58d9b4d459-zlw98\" (UID: \"505b718c-f2ab-4f6d-892a-ed614bc2335d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98" Oct 31 19:38:32.439688 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:32.439650 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/505b718c-f2ab-4f6d-892a-ed614bc2335d-openshift-service-ca\") pod \"maas-api-e2e-mt-db61066b-58d9b4d459-zlw98\" (UID: \"505b718c-f2ab-4f6d-892a-ed614bc2335d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98" Oct 31 19:38:32.444101 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:32.444058 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/505b718c-f2ab-4f6d-892a-ed614bc2335d-maas-api-tls\") pod \"maas-api-e2e-mt-db61066b-58d9b4d459-zlw98\" (UID: \"505b718c-f2ab-4f6d-892a-ed614bc2335d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98" Oct 31 19:38:32.445709 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:32.445676 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/505b718c-f2ab-4f6d-892a-ed614bc2335d-openshift-service-ca\") pod \"maas-api-e2e-mt-db61066b-58d9b4d459-zlw98\" (UID: \"505b718c-f2ab-4f6d-892a-ed614bc2335d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98" Oct 31 19:38:32.449790 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:32.449767 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/505b718c-f2ab-4f6d-892a-ed614bc2335d-metrics-certs\") pod \"maas-api-e2e-mt-db61066b-58d9b4d459-zlw98\" (UID: \"505b718c-f2ab-4f6d-892a-ed614bc2335d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98" Oct 31 19:38:32.470983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:32.460797 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/505b718c-f2ab-4f6d-892a-ed614bc2335d-maas-api-tls\") pod \"maas-api-e2e-mt-db61066b-58d9b4d459-zlw98\" (UID: \"505b718c-f2ab-4f6d-892a-ed614bc2335d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98" Oct 31 19:38:32.470983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:32.466556 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz6ht\" (UniqueName: \"kubernetes.io/projected/505b718c-f2ab-4f6d-892a-ed614bc2335d-kube-api-access-lz6ht\") pod \"maas-api-e2e-mt-db61066b-58d9b4d459-zlw98\" (UID: \"505b718c-f2ab-4f6d-892a-ed614bc2335d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98" Oct 31 19:38:32.573817 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:32.573783 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98" Oct 31 19:38:32.973932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:32.968737 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98"] Oct 31 19:38:33.167317 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:33.167223 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98" event={"ID":"505b718c-f2ab-4f6d-892a-ed614bc2335d","Type":"ContainerStarted","Data":"9e626ff3710b54853fb9f1ac6192c6ebdd47d2c7dc5385a31b20758ad51b10d6"} Oct 31 19:38:34.178768 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:34.178728 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98" event={"ID":"505b718c-f2ab-4f6d-892a-ed614bc2335d","Type":"ContainerStarted","Data":"42eef546a273aabac292f803e28b8ec7af7fb7361081b16f2ca1765fdbd10232"} Oct 31 19:38:34.179202 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:34.178791 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98" Oct 31 19:38:34.201575 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:34.201529 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98" podStartSLOduration=1.6901054229999999 podStartE2EDuration="2.20151477s" totalImagesPullingTime="511.409347ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:34.200260212 +0000 UTC m=+1148.563080581" watchObservedRunningTime="2026-10-31 19:38:34.20151477 +0000 UTC m=+1148.564335142" Oct 31 19:38:40.197697 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:40.197665 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98" Oct 31 19:38:42.713823 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:38:42.713786 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:38:44.053942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:44.053220 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58c55970-lql9j"] Oct 31 19:38:44.060756 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:44.060728 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58c55970-lql9j"] Oct 31 19:38:45.761448 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:45.761411 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a666bdeb-b0ad-450e-b44a-a05ab6835ace" path="/var/lib/kubelet/pods/a666bdeb-b0ad-450e-b44a-a05ab6835ace/volumes" Oct 31 19:38:46.387965 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:46.372813 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-2q285"] Oct 31 19:38:46.388181 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:46.388061 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-2q285" Oct 31 19:38:46.394185 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:46.394147 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-2q285"] Oct 31 19:38:46.395018 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:46.394984 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:38:46.395247 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:46.395150 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:38:46.446016 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:46.445972 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwqk8\" (UniqueName: \"kubernetes.io/projected/99fa69c0-fb59-4711-9d25-2813eeaba33b-kube-api-access-qwqk8\") pod \"maas-api-revoke-keys-e2e-mt-67adab47-2q285\" (UID: \"99fa69c0-fb59-4711-9d25-2813eeaba33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-2q285" Oct 31 19:38:46.446229 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:46.446089 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99fa69c0-fb59-4711-9d25-2813eeaba33b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-67adab47-2q285\" (UID: \"99fa69c0-fb59-4711-9d25-2813eeaba33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-2q285" Oct 31 19:38:46.446298 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:46.446225 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99fa69c0-fb59-4711-9d25-2813eeaba33b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-67adab47-2q285\" (UID: \"99fa69c0-fb59-4711-9d25-2813eeaba33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-2q285" Oct 31 19:38:46.547675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:46.547635 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99fa69c0-fb59-4711-9d25-2813eeaba33b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-67adab47-2q285\" (UID: \"99fa69c0-fb59-4711-9d25-2813eeaba33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-2q285" Oct 31 19:38:46.547898 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:46.547827 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwqk8\" (UniqueName: \"kubernetes.io/projected/99fa69c0-fb59-4711-9d25-2813eeaba33b-kube-api-access-qwqk8\") pod \"maas-api-revoke-keys-e2e-mt-67adab47-2q285\" (UID: \"99fa69c0-fb59-4711-9d25-2813eeaba33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-2q285" Oct 31 19:38:46.547898 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:46.547889 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99fa69c0-fb59-4711-9d25-2813eeaba33b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-67adab47-2q285\" (UID: \"99fa69c0-fb59-4711-9d25-2813eeaba33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-2q285" Oct 31 19:38:46.548599 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:46.548575 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99fa69c0-fb59-4711-9d25-2813eeaba33b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-67adab47-2q285\" (UID: \"99fa69c0-fb59-4711-9d25-2813eeaba33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-2q285" Oct 31 19:38:46.550408 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:46.550378 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99fa69c0-fb59-4711-9d25-2813eeaba33b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-67adab47-2q285\" (UID: \"99fa69c0-fb59-4711-9d25-2813eeaba33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-2q285" Oct 31 19:38:46.560807 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:46.560781 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwqk8\" (UniqueName: \"kubernetes.io/projected/99fa69c0-fb59-4711-9d25-2813eeaba33b-kube-api-access-qwqk8\") pod \"maas-api-revoke-keys-e2e-mt-67adab47-2q285\" (UID: \"99fa69c0-fb59-4711-9d25-2813eeaba33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-2q285" Oct 31 19:38:46.725807 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:46.725710 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-2q285" Oct 31 19:38:47.021801 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:38:47.021761 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99fa69c0_fb59_4711_9d25_2813eeaba33b.slice/crio-28a1f520846914e331a93741dfdb4592796bda9e628b048b995d0b59b0df9f35 WatchSource:0}: Error finding container 28a1f520846914e331a93741dfdb4592796bda9e628b048b995d0b59b0df9f35: Status 404 returned error can't find the container with id 28a1f520846914e331a93741dfdb4592796bda9e628b048b995d0b59b0df9f35 Oct 31 19:38:47.022723 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:47.022697 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-2q285"] Oct 31 19:38:47.475936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:47.474827 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-2q285" event={"ID":"99fa69c0-fb59-4711-9d25-2813eeaba33b","Type":"ContainerStarted","Data":"a8f55653b7c19e7a81939f9fdf790f36e4d31a8579515a9222f684c28f4d205b"} Oct 31 19:38:47.475936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:47.474895 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-2q285" event={"ID":"99fa69c0-fb59-4711-9d25-2813eeaba33b","Type":"ContainerStarted","Data":"28a1f520846914e331a93741dfdb4592796bda9e628b048b995d0b59b0df9f35"} Oct 31 19:38:47.508040 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:47.507971 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-2q285" podStartSLOduration=1.507951132 podStartE2EDuration="1.507951132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:47.496112265 +0000 UTC m=+1161.858932638" watchObservedRunningTime="2026-10-31 19:38:47.507951132 +0000 UTC m=+1161.870771517" Oct 31 19:38:48.571597 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:48.571550 4910 generic.go:356] "Generic (PLEG): container finished" podID="99fa69c0-fb59-4711-9d25-2813eeaba33b" containerID="a8f55653b7c19e7a81939f9fdf790f36e4d31a8579515a9222f684c28f4d205b" exitCode=0 Oct 31 19:38:48.572129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:48.571835 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-2q285" event={"ID":"99fa69c0-fb59-4711-9d25-2813eeaba33b","Type":"ContainerDied","Data":"a8f55653b7c19e7a81939f9fdf790f36e4d31a8579515a9222f684c28f4d205b"} Oct 31 19:38:50.053446 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:50.053415 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-2q285" Oct 31 19:38:50.286000 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:50.263661 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99fa69c0-fb59-4711-9d25-2813eeaba33b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99fa69c0-fb59-4711-9d25-2813eeaba33b-maas-api-openshift-ca\") pod \"99fa69c0-fb59-4711-9d25-2813eeaba33b\" (UID: \"99fa69c0-fb59-4711-9d25-2813eeaba33b\") " Oct 31 19:38:50.286000 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:50.263784 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99fa69c0-fb59-4711-9d25-2813eeaba33b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99fa69c0-fb59-4711-9d25-2813eeaba33b-maas-api-xks-ca\") pod \"99fa69c0-fb59-4711-9d25-2813eeaba33b\" (UID: \"99fa69c0-fb59-4711-9d25-2813eeaba33b\") " Oct 31 19:38:50.295160 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:50.295102 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99fa69c0-fb59-4711-9d25-2813eeaba33b-maas-api-openshift-ca" pod "99fa69c0-fb59-4711-9d25-2813eeaba33b" (UID: "99fa69c0-fb59-4711-9d25-2813eeaba33b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:38:50.303975 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:50.295445 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99fa69c0-fb59-4711-9d25-2813eeaba33b-kube-api-access-qwqk8\" (UniqueName: \"kubernetes.io/projected/99fa69c0-fb59-4711-9d25-2813eeaba33b-kube-api-access-qwqk8\") pod \"99fa69c0-fb59-4711-9d25-2813eeaba33b\" (UID: \"99fa69c0-fb59-4711-9d25-2813eeaba33b\") " Oct 31 19:38:50.303975 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:50.296371 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99fa69c0-fb59-4711-9d25-2813eeaba33b-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:38:50.328131 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:50.326072 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99fa69c0-fb59-4711-9d25-2813eeaba33b-maas-api-xks-ca" pod "99fa69c0-fb59-4711-9d25-2813eeaba33b" (UID: "99fa69c0-fb59-4711-9d25-2813eeaba33b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:38:50.413415 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:50.413320 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99fa69c0-fb59-4711-9d25-2813eeaba33b-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:38:50.417350 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:50.417303 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99fa69c0-fb59-4711-9d25-2813eeaba33b-kube-api-access-qwqk8" pod "99fa69c0-fb59-4711-9d25-2813eeaba33b" (UID: "99fa69c0-fb59-4711-9d25-2813eeaba33b"). InnerVolumeSpecName "kube-api-access-qwqk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:38:50.515670 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:50.515631 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwqk8\" (UniqueName: \"kubernetes.io/projected/99fa69c0-fb59-4711-9d25-2813eeaba33b-kube-api-access-qwqk8\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:38:50.825981 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:50.807518 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-2q285" event={"ID":"99fa69c0-fb59-4711-9d25-2813eeaba33b","Type":"ContainerDied","Data":"28a1f520846914e331a93741dfdb4592796bda9e628b048b995d0b59b0df9f35"} Oct 31 19:38:50.825981 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:50.807586 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28a1f520846914e331a93741dfdb4592796bda9e628b048b995d0b59b0df9f35" Oct 31 19:38:50.825981 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:50.807652 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-2q285" Oct 31 19:38:52.817061 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:52.817005 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:53.118348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:53.118221 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:53.121943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:53.121428 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:53.121943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:53.121733 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:38:53.232792 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:53.232742 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 19:38:53.235360 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:53.235020 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:38:53.259961 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:53.259900 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:53.277473 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:53.277428 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:53.298795 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:53.298753 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:53.340502 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:53.340452 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:38:53.623938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:53.620440 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:53.803937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:53.798022 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:53.827931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:53.823994 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:53.872938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:53.872189 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:38:54.171028 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.170984 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:38:54.171632 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.171601 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:38:54.178615 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.178591 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:38:54.263936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.214710 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:38:54.263936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.263053 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:38:54.275947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.237974 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:54.275947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.237995 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 19:38:54.275947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.265504 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:38:54.275947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.265556 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:38:54.275947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.267026 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:38:54.278462 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.237926 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:54.280062 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.280026 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:54.280584 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.280251 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:54.291695 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.284138 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:38:54.292459 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.292437 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:38:54.322543 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.302529 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:54.322543 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.302971 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:38:54.322543 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.303730 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:54.389475 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.351015 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:54.389475 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.352424 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:38:54.566073 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.566019 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:54.621306 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.621256 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:54.623020 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.622993 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:38:54.692234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.692193 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:38:54.779935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.770739 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:38:54.808466 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.800038 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:54.808466 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.801555 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:38:54.808466 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.803332 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:38:54.837893 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.837783 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:38:54.994047 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.994016 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:38:54.994307 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.994291 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:38:54.997282 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:54.997256 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:38:55.018943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:55.018046 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:38:55.018943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:55.018136 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:38:55.066287 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:55.064928 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:38:55.111358 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:55.111271 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:38:55.118480 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:55.118447 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:38:55.202269 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:55.202235 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:38:55.210319 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:55.210288 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c54dc7ab-r47q6"] Oct 31 19:38:55.216062 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:55.216035 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c54dc7ab-r47q6"] Oct 31 19:38:55.841872 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:55.840610 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="623cd320-b619-489a-bff1-c63ec2795ecd" path="/var/lib/kubelet/pods/623cd320-b619-489a-bff1-c63ec2795ecd/volumes" Oct 31 19:38:56.084368 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:56.070967 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-753b8133-xnnxd"] Oct 31 19:38:56.106365 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:56.106241 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-753b8133-xnnxd"] Oct 31 19:38:57.764708 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:57.764670 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b5dfe21-5d45-487b-9788-ce7e303d18d9" path="/var/lib/kubelet/pods/7b5dfe21-5d45-487b-9788-ce7e303d18d9/volumes" Oct 31 19:38:58.591004 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:58.590964 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0576ba46-rclk7"] Oct 31 19:38:58.592048 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:58.592027 4910 cpu_manager.go:414] "containerMap: removing container" podUID="99fa69c0-fb59-4711-9d25-2813eeaba33b" containerName="revoke-keys" Oct 31 19:38:58.592130 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:58.592048 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99fa69c0-fb59-4711-9d25-2813eeaba33b" containerName="revoke-keys" Oct 31 19:38:58.592230 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:58.592213 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="99fa69c0-fb59-4711-9d25-2813eeaba33b" containerName="revoke-keys" Oct 31 19:38:58.598223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:58.598193 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0576ba46-rclk7" Oct 31 19:38:58.604818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:58.604783 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0576ba46-rclk7"] Oct 31 19:38:58.605263 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:58.605244 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:38:58.605456 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:58.605443 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:38:58.715002 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:58.707812 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvcvd\" (UniqueName: \"kubernetes.io/projected/4d191ec1-875b-4dc3-a3f9-e896d53dc00f-kube-api-access-jvcvd\") pod \"maas-api-revoke-sub-0576ba46-rclk7\" (UID: \"4d191ec1-875b-4dc3-a3f9-e896d53dc00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0576ba46-rclk7" Oct 31 19:38:58.715002 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:58.708045 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d191ec1-875b-4dc3-a3f9-e896d53dc00f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0576ba46-rclk7\" (UID: \"4d191ec1-875b-4dc3-a3f9-e896d53dc00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0576ba46-rclk7" Oct 31 19:38:58.715002 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:58.708138 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d191ec1-875b-4dc3-a3f9-e896d53dc00f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0576ba46-rclk7\" (UID: \"4d191ec1-875b-4dc3-a3f9-e896d53dc00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0576ba46-rclk7" Oct 31 19:38:58.862316 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:58.862216 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvcvd\" (UniqueName: \"kubernetes.io/projected/4d191ec1-875b-4dc3-a3f9-e896d53dc00f-kube-api-access-jvcvd\") pod \"maas-api-revoke-sub-0576ba46-rclk7\" (UID: \"4d191ec1-875b-4dc3-a3f9-e896d53dc00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0576ba46-rclk7" Oct 31 19:38:58.894935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:58.867474 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d191ec1-875b-4dc3-a3f9-e896d53dc00f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0576ba46-rclk7\" (UID: \"4d191ec1-875b-4dc3-a3f9-e896d53dc00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0576ba46-rclk7" Oct 31 19:38:58.944954 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:58.944904 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d191ec1-875b-4dc3-a3f9-e896d53dc00f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0576ba46-rclk7\" (UID: \"4d191ec1-875b-4dc3-a3f9-e896d53dc00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0576ba46-rclk7" Oct 31 19:38:58.945830 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:58.945800 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d191ec1-875b-4dc3-a3f9-e896d53dc00f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0576ba46-rclk7\" (UID: \"4d191ec1-875b-4dc3-a3f9-e896d53dc00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0576ba46-rclk7" Oct 31 19:38:58.992742 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:58.992705 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvcvd\" (UniqueName: \"kubernetes.io/projected/4d191ec1-875b-4dc3-a3f9-e896d53dc00f-kube-api-access-jvcvd\") pod \"maas-api-revoke-sub-0576ba46-rclk7\" (UID: \"4d191ec1-875b-4dc3-a3f9-e896d53dc00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0576ba46-rclk7" Oct 31 19:38:59.000413 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:59.000382 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d191ec1-875b-4dc3-a3f9-e896d53dc00f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0576ba46-rclk7\" (UID: \"4d191ec1-875b-4dc3-a3f9-e896d53dc00f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0576ba46-rclk7" Oct 31 19:38:59.012126 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:59.011498 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0576ba46-rclk7" Oct 31 19:38:59.055899 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:59.055846 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1858e1c-p5zt6"] Oct 31 19:38:59.065371 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:59.065324 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1858e1c-p5zt6"] Oct 31 19:38:59.326483 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:38:59.326448 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d191ec1_875b_4dc3_a3f9_e896d53dc00f.slice/crio-9e5138d97bd76ad1c24510770911a789f725705050824bbb83fedd34c035a50b WatchSource:0}: Error finding container 9e5138d97bd76ad1c24510770911a789f725705050824bbb83fedd34c035a50b: Status 404 returned error can't find the container with id 9e5138d97bd76ad1c24510770911a789f725705050824bbb83fedd34c035a50b Oct 31 19:38:59.329058 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:59.328689 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0576ba46-rclk7"] Oct 31 19:38:59.632646 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:59.632607 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0576ba46-rclk7" event={"ID":"4d191ec1-875b-4dc3-a3f9-e896d53dc00f","Type":"ContainerStarted","Data":"9e5138d97bd76ad1c24510770911a789f725705050824bbb83fedd34c035a50b"} Oct 31 19:38:59.766296 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:38:59.766256 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0fc3b05-0e60-417e-bd39-ce23ea580292" path="/var/lib/kubelet/pods/a0fc3b05-0e60-417e-bd39-ce23ea580292/volumes" Oct 31 19:39:00.648699 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:00.648656 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0576ba46-rclk7" event={"ID":"4d191ec1-875b-4dc3-a3f9-e896d53dc00f","Type":"ContainerStarted","Data":"a53cdf1855d4041730daa3e0be768c2d68a4e8b761e18cf3666c31766c1506cf"} Oct 31 19:39:00.673320 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:00.673262 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0576ba46-rclk7" podStartSLOduration=2.673248756 podStartE2EDuration="2.673248756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:00.66764407 +0000 UTC m=+1175.030464479" watchObservedRunningTime="2026-10-31 19:39:00.673248756 +0000 UTC m=+1175.036069128" Oct 31 19:39:01.660690 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:01.660653 4910 generic.go:356] "Generic (PLEG): container finished" podID="4d191ec1-875b-4dc3-a3f9-e896d53dc00f" containerID="a53cdf1855d4041730daa3e0be768c2d68a4e8b761e18cf3666c31766c1506cf" exitCode=0 Oct 31 19:39:01.661277 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:01.660743 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0576ba46-rclk7" event={"ID":"4d191ec1-875b-4dc3-a3f9-e896d53dc00f","Type":"ContainerDied","Data":"a53cdf1855d4041730daa3e0be768c2d68a4e8b761e18cf3666c31766c1506cf"} Oct 31 19:39:01.885595 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:01.885557 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2"] Oct 31 19:39:01.891371 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:01.890988 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:01.912895 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:01.912766 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-ae2e60-openshift-default-dockercfg-ts6pq\"" Oct 31 19:39:01.913526 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:01.913399 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2"] Oct 31 19:39:01.987296 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:01.987172 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:01.987296 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:01.987226 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:01.987296 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:01.987266 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:01.987621 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:01.987356 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:01.987621 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:01.987401 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:01.987621 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:01.987514 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/949b61a5-f07e-47c1-b360-a3bfde012f42-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:01.987621 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:01.987547 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:01.987621 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:01.987568 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zz8n\" (UniqueName: \"kubernetes.io/projected/949b61a5-f07e-47c1-b360-a3bfde012f42-kube-api-access-8zz8n\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:01.987885 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:01.987631 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:02.039930 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:02.036283 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d470d6-95bd6"] Oct 31 19:39:02.055959 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:02.054524 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08d470d6-95bd6"] Oct 31 19:39:02.092933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:02.092575 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:02.092933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:02.092670 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:02.092933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:02.092846 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/949b61a5-f07e-47c1-b360-a3bfde012f42-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:02.092933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:02.092893 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:02.093317 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:02.092945 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zz8n\" (UniqueName: \"kubernetes.io/projected/949b61a5-f07e-47c1-b360-a3bfde012f42-kube-api-access-8zz8n\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:02.093317 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:02.093047 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:02.093317 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:02.093089 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:02.093317 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:02.093138 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:02.093317 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:02.093186 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:02.116945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:02.109208 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:02.116945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:02.109790 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:02.116945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:02.110479 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/949b61a5-f07e-47c1-b360-a3bfde012f42-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:02.116945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:02.110730 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:02.127564 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:02.126633 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:02.156939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:02.155164 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:02.180983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:02.178906 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zz8n\" (UniqueName: \"kubernetes.io/projected/949b61a5-f07e-47c1-b360-a3bfde012f42-kube-api-access-8zz8n\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:02.230734 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:02.230701 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:02.230992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:02.230706 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:02.546823 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:02.546713 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:02.951375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:02.948269 4910 patch_prober.go:29] interesting pod/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": read tcp 10.132.0.2:42374->10.132.0.142:15021: read: connection reset by peer" start-of-body= Oct 31 19:39:02.951375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:02.948345 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" podUID="790bd78d-4525-4587-9b6f-30af5841a88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": read tcp 10.132.0.2:42374->10.132.0.142:15021: read: connection reset by peer" Oct 31 19:39:02.996992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:02.996962 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2"] Oct 31 19:39:03.009784 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:39:03.009747 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod949b61a5_f07e_47c1_b360_a3bfde012f42.slice/crio-cb27839590dea226c7fb3b7ea54a3d222636adda71df82a43c0c2b02b5909f7e WatchSource:0}: Error finding container cb27839590dea226c7fb3b7ea54a3d222636adda71df82a43c0c2b02b5909f7e: Status 404 returned error can't find the container with id cb27839590dea226c7fb3b7ea54a3d222636adda71df82a43c0c2b02b5909f7e Oct 31 19:39:03.013072 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.013035 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:39:03.013203 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.013108 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:39:03.013203 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.013141 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:39:03.013987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.013968 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0576ba46-rclk7" Oct 31 19:39:03.058436 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.058403 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d191ec1-875b-4dc3-a3f9-e896d53dc00f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d191ec1-875b-4dc3-a3f9-e896d53dc00f-maas-api-openshift-ca\") pod \"4d191ec1-875b-4dc3-a3f9-e896d53dc00f\" (UID: \"4d191ec1-875b-4dc3-a3f9-e896d53dc00f\") " Oct 31 19:39:03.058608 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.058590 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d191ec1-875b-4dc3-a3f9-e896d53dc00f-kube-api-access-jvcvd\" (UniqueName: \"kubernetes.io/projected/4d191ec1-875b-4dc3-a3f9-e896d53dc00f-kube-api-access-jvcvd\") pod \"4d191ec1-875b-4dc3-a3f9-e896d53dc00f\" (UID: \"4d191ec1-875b-4dc3-a3f9-e896d53dc00f\") " Oct 31 19:39:03.058753 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.058736 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d191ec1-875b-4dc3-a3f9-e896d53dc00f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d191ec1-875b-4dc3-a3f9-e896d53dc00f-maas-api-xks-ca\") pod \"4d191ec1-875b-4dc3-a3f9-e896d53dc00f\" (UID: \"4d191ec1-875b-4dc3-a3f9-e896d53dc00f\") " Oct 31 19:39:03.060137 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.060106 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d191ec1-875b-4dc3-a3f9-e896d53dc00f-maas-api-openshift-ca" pod "4d191ec1-875b-4dc3-a3f9-e896d53dc00f" (UID: "4d191ec1-875b-4dc3-a3f9-e896d53dc00f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:39:03.064462 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.064424 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d191ec1-875b-4dc3-a3f9-e896d53dc00f-kube-api-access-jvcvd" pod "4d191ec1-875b-4dc3-a3f9-e896d53dc00f" (UID: "4d191ec1-875b-4dc3-a3f9-e896d53dc00f"). InnerVolumeSpecName "kube-api-access-jvcvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:39:03.069939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.066983 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d191ec1-875b-4dc3-a3f9-e896d53dc00f-maas-api-xks-ca" pod "4d191ec1-875b-4dc3-a3f9-e896d53dc00f" (UID: "4d191ec1-875b-4dc3-a3f9-e896d53dc00f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:39:03.161586 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.161538 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d191ec1-875b-4dc3-a3f9-e896d53dc00f-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:03.161586 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.161579 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvcvd\" (UniqueName: \"kubernetes.io/projected/4d191ec1-875b-4dc3-a3f9-e896d53dc00f-kube-api-access-jvcvd\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:03.161800 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.161594 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d191ec1-875b-4dc3-a3f9-e896d53dc00f-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:03.708091 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.708052 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" event={"ID":"949b61a5-f07e-47c1-b360-a3bfde012f42","Type":"ContainerStarted","Data":"77cc12100e7999d67458362233f0ea318c168ef7b636d85a2cdef5eafe159689"} Oct 31 19:39:03.708296 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.708112 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" event={"ID":"949b61a5-f07e-47c1-b360-a3bfde012f42","Type":"ContainerStarted","Data":"cb27839590dea226c7fb3b7ea54a3d222636adda71df82a43c0c2b02b5909f7e"} Oct 31 19:39:03.709967 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.709940 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0576ba46-rclk7" event={"ID":"4d191ec1-875b-4dc3-a3f9-e896d53dc00f","Type":"ContainerDied","Data":"9e5138d97bd76ad1c24510770911a789f725705050824bbb83fedd34c035a50b"} Oct 31 19:39:03.710090 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.709988 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e5138d97bd76ad1c24510770911a789f725705050824bbb83fedd34c035a50b" Oct 31 19:39:03.710090 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.709952 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0576ba46-rclk7" Oct 31 19:39:03.712474 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.712442 4910 generic.go:356] "Generic (PLEG): container finished" podID="790bd78d-4525-4587-9b6f-30af5841a88a" containerID="db127c0d1f3a7531ff62c648fe12abfcc64371e36d96e7f50f8bcfeadc1ff6a7" exitCode=0 Oct 31 19:39:03.712575 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.712488 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" event={"ID":"790bd78d-4525-4587-9b6f-30af5841a88a","Type":"ContainerDied","Data":"db127c0d1f3a7531ff62c648fe12abfcc64371e36d96e7f50f8bcfeadc1ff6a7"} Oct 31 19:39:03.713012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.712989 4910 scope.go:122] "RemoveContainer" containerID="db127c0d1f3a7531ff62c648fe12abfcc64371e36d96e7f50f8bcfeadc1ff6a7" Oct 31 19:39:03.715041 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.715009 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:39:03.715116 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.715099 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:39:03.715161 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.715145 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:39:03.751425 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.751372 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" podStartSLOduration=2.751353857 podStartE2EDuration="2.751353857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:03.745276669 +0000 UTC m=+1178.108097042" watchObservedRunningTime="2026-10-31 19:39:03.751353857 +0000 UTC m=+1178.114174230" Oct 31 19:39:03.761932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:03.761890 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dde4aa8-330b-4bdd-93c3-58ae26d99dfe" path="/var/lib/kubelet/pods/4dde4aa8-330b-4bdd-93c3-58ae26d99dfe/volumes" Oct 31 19:39:04.472473 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.472386 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9"] Oct 31 19:39:04.473073 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.472636 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" podUID="012ba66a-ce6f-4261-9516-e8f0aa8ff815" containerName="istio-proxy" containerID="cri-o://f3dfb43990ecb35499512e9cf0b6372b1b6a9005d56fa8b7666d39f934e48b2b" gracePeriod=30 Oct 31 19:39:04.481195 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.480881 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf"] Oct 31 19:39:04.481195 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.481177 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" podUID="09156e2c-03fe-4f93-af3a-4449f61f1e2c" containerName="istio-proxy" containerID="cri-o://1ca447bd12143f53b0b0c4b345c1fe9e80305c79c26614cb1d824b359454ec3a" gracePeriod=30 Oct 31 19:39:04.489979 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.489078 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z"] Oct 31 19:39:04.489979 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.489296 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" podUID="336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52" containerName="istio-proxy" containerID="cri-o://9c01437dae8454927a1d8cf486034b36fbb6999e0bd0d8a8793e664be8078ca3" gracePeriod=30 Oct 31 19:39:04.550443 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.549970 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:04.556816 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.556780 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:04.659751 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.659719 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d16ace94-jqddp"] Oct 31 19:39:04.660507 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.660490 4910 cpu_manager.go:414] "containerMap: removing container" podUID="4d191ec1-875b-4dc3-a3f9-e896d53dc00f" containerName="revoke-keys" Oct 31 19:39:04.660563 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.660509 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d191ec1-875b-4dc3-a3f9-e896d53dc00f" containerName="revoke-keys" Oct 31 19:39:04.660648 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.660639 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d191ec1-875b-4dc3-a3f9-e896d53dc00f" containerName="revoke-keys" Oct 31 19:39:04.664538 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.664512 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16ace94-jqddp" Oct 31 19:39:04.675398 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.675368 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d16ace94-jqddp"] Oct 31 19:39:04.743163 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.743066 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" event={"ID":"790bd78d-4525-4587-9b6f-30af5841a88a","Type":"ContainerStarted","Data":"046c42181792f3e97062e05ffee400dd07f1509f7754c952cadb1a8be35e1b9d"} Oct 31 19:39:04.743449 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.743429 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:04.744456 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.744429 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:39:04.755616 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.755587 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ebff398-c9da-4829-910b-497908ea7feb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d16ace94-jqddp\" (UID: \"0ebff398-c9da-4829-910b-497908ea7feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16ace94-jqddp" Oct 31 19:39:04.756481 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.756454 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d7mn\" (UniqueName: \"kubernetes.io/projected/0ebff398-c9da-4829-910b-497908ea7feb-kube-api-access-2d7mn\") pod \"maas-api-revoke-sub-d16ace94-jqddp\" (UID: \"0ebff398-c9da-4829-910b-497908ea7feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16ace94-jqddp" Oct 31 19:39:04.756595 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.756563 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ebff398-c9da-4829-910b-497908ea7feb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d16ace94-jqddp\" (UID: \"0ebff398-c9da-4829-910b-497908ea7feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16ace94-jqddp" Oct 31 19:39:04.762530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.762471 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:39:04.762619 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.762543 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:39:04.781281 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.781249 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:39:04.781476 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.781448 4910 patch_prober.go:29] interesting pod/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 19:39:04.781686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.781656 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" podUID="790bd78d-4525-4587-9b6f-30af5841a88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 19:39:04.861249 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.861202 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ebff398-c9da-4829-910b-497908ea7feb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d16ace94-jqddp\" (UID: \"0ebff398-c9da-4829-910b-497908ea7feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16ace94-jqddp" Oct 31 19:39:04.861457 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.861415 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d7mn\" (UniqueName: \"kubernetes.io/projected/0ebff398-c9da-4829-910b-497908ea7feb-kube-api-access-2d7mn\") pod \"maas-api-revoke-sub-d16ace94-jqddp\" (UID: \"0ebff398-c9da-4829-910b-497908ea7feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16ace94-jqddp" Oct 31 19:39:04.861561 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.861542 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ebff398-c9da-4829-910b-497908ea7feb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d16ace94-jqddp\" (UID: \"0ebff398-c9da-4829-910b-497908ea7feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16ace94-jqddp" Oct 31 19:39:04.869776 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.868948 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ebff398-c9da-4829-910b-497908ea7feb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d16ace94-jqddp\" (UID: \"0ebff398-c9da-4829-910b-497908ea7feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16ace94-jqddp" Oct 31 19:39:04.894198 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.891161 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d7mn\" (UniqueName: \"kubernetes.io/projected/0ebff398-c9da-4829-910b-497908ea7feb-kube-api-access-2d7mn\") pod \"maas-api-revoke-sub-d16ace94-jqddp\" (UID: \"0ebff398-c9da-4829-910b-497908ea7feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16ace94-jqddp" Oct 31 19:39:04.902978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:04.901089 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ebff398-c9da-4829-910b-497908ea7feb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d16ace94-jqddp\" (UID: \"0ebff398-c9da-4829-910b-497908ea7feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16ace94-jqddp" Oct 31 19:39:05.000115 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:05.000028 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16ace94-jqddp" Oct 31 19:39:05.060699 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:05.060669 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dd2ce39-pr7sr"] Oct 31 19:39:05.080054 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:05.079928 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dd2ce39-pr7sr"] Oct 31 19:39:05.766429 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:05.765985 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3094a11-6c88-4340-b63b-03d19ad814dd" path="/var/lib/kubelet/pods/f3094a11-6c88-4340-b63b-03d19ad814dd/volumes" Oct 31 19:39:05.781812 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:05.781782 4910 patch_prober.go:29] interesting pod/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 19:39:05.781995 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:05.781826 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" podUID="790bd78d-4525-4587-9b6f-30af5841a88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 19:39:05.792600 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:39:05.792572 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ebff398_c9da_4829_910b_497908ea7feb.slice/crio-8b977f009e8f8b63ec3de968a62cbbe30542fa37528c7d2663a368cb9a1cb842 WatchSource:0}: Error finding container 8b977f009e8f8b63ec3de968a62cbbe30542fa37528c7d2663a368cb9a1cb842: Status 404 returned error can't find the container with id 8b977f009e8f8b63ec3de968a62cbbe30542fa37528c7d2663a368cb9a1cb842 Oct 31 19:39:05.793150 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:05.793130 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d16ace94-jqddp"] Oct 31 19:39:06.790979 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:06.790940 4910 patch_prober.go:29] interesting pod/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 19:39:06.814547 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:06.814503 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" podUID="790bd78d-4525-4587-9b6f-30af5841a88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 19:39:06.904237 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:06.900556 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16ace94-jqddp" event={"ID":"0ebff398-c9da-4829-910b-497908ea7feb","Type":"ContainerStarted","Data":"1cbe3247361f3b913c24c22b33982f5689e62d63a00b4ace7a7179086f84ec33"} Oct 31 19:39:06.904237 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:06.900619 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16ace94-jqddp" event={"ID":"0ebff398-c9da-4829-910b-497908ea7feb","Type":"ContainerStarted","Data":"8b977f009e8f8b63ec3de968a62cbbe30542fa37528c7d2663a368cb9a1cb842"} Oct 31 19:39:06.972107 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:06.972008 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16ace94-jqddp" podStartSLOduration=2.971986246 podStartE2EDuration="2.971986246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:06.95370934 +0000 UTC m=+1181.316529713" watchObservedRunningTime="2026-10-31 19:39:06.971986246 +0000 UTC m=+1181.334806620" Oct 31 19:39:07.786947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:07.786698 4910 patch_prober.go:29] interesting pod/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 19:39:07.795541 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:07.787266 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" podUID="790bd78d-4525-4587-9b6f-30af5841a88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 19:39:07.913957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:07.911165 4910 generic.go:356] "Generic (PLEG): container finished" podID="0ebff398-c9da-4829-910b-497908ea7feb" containerID="1cbe3247361f3b913c24c22b33982f5689e62d63a00b4ace7a7179086f84ec33" exitCode=0 Oct 31 19:39:07.913957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:07.911222 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16ace94-jqddp" event={"ID":"0ebff398-c9da-4829-910b-497908ea7feb","Type":"ContainerDied","Data":"1cbe3247361f3b913c24c22b33982f5689e62d63a00b4ace7a7179086f84ec33"} Oct 31 19:39:08.782295 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:08.782265 4910 patch_prober.go:29] interesting pod/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 19:39:08.782529 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:08.782508 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" podUID="790bd78d-4525-4587-9b6f-30af5841a88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 19:39:09.273202 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.273174 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16ace94-jqddp" Oct 31 19:39:09.319781 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.319746 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ebff398-c9da-4829-910b-497908ea7feb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ebff398-c9da-4829-910b-497908ea7feb-maas-api-xks-ca\") pod \"0ebff398-c9da-4829-910b-497908ea7feb\" (UID: \"0ebff398-c9da-4829-910b-497908ea7feb\") " Oct 31 19:39:09.321225 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.321200 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ebff398-c9da-4829-910b-497908ea7feb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ebff398-c9da-4829-910b-497908ea7feb-maas-api-openshift-ca\") pod \"0ebff398-c9da-4829-910b-497908ea7feb\" (UID: \"0ebff398-c9da-4829-910b-497908ea7feb\") " Oct 31 19:39:09.321961 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.321941 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ebff398-c9da-4829-910b-497908ea7feb-kube-api-access-2d7mn\" (UniqueName: \"kubernetes.io/projected/0ebff398-c9da-4829-910b-497908ea7feb-kube-api-access-2d7mn\") pod \"0ebff398-c9da-4829-910b-497908ea7feb\" (UID: \"0ebff398-c9da-4829-910b-497908ea7feb\") " Oct 31 19:39:09.322515 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.321700 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ebff398-c9da-4829-910b-497908ea7feb-maas-api-openshift-ca" pod "0ebff398-c9da-4829-910b-497908ea7feb" (UID: "0ebff398-c9da-4829-910b-497908ea7feb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:39:09.323477 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.323452 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ebff398-c9da-4829-910b-497908ea7feb-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:09.357148 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.357109 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ebff398-c9da-4829-910b-497908ea7feb-maas-api-xks-ca" pod "0ebff398-c9da-4829-910b-497908ea7feb" (UID: "0ebff398-c9da-4829-910b-497908ea7feb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:39:09.362273 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.362243 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ebff398-c9da-4829-910b-497908ea7feb-kube-api-access-2d7mn" pod "0ebff398-c9da-4829-910b-497908ea7feb" (UID: "0ebff398-c9da-4829-910b-497908ea7feb"). InnerVolumeSpecName "kube-api-access-2d7mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:39:09.442968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.442392 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ebff398-c9da-4829-910b-497908ea7feb-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:09.442968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.442443 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d7mn\" (UniqueName: \"kubernetes.io/projected/0ebff398-c9da-4829-910b-497908ea7feb-kube-api-access-2d7mn\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:09.884703 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.884664 4910 patch_prober.go:29] interesting pod/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 19:39:09.884857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.884733 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" podUID="790bd78d-4525-4587-9b6f-30af5841a88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 19:39:09.969594 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.969516 4910 generic.go:356] "Generic (PLEG): container finished" podID="012ba66a-ce6f-4261-9516-e8f0aa8ff815" containerID="f3dfb43990ecb35499512e9cf0b6372b1b6a9005d56fa8b7666d39f934e48b2b" exitCode=0 Oct 31 19:39:09.969893 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.969570 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" event={"ID":"012ba66a-ce6f-4261-9516-e8f0aa8ff815","Type":"ContainerDied","Data":"f3dfb43990ecb35499512e9cf0b6372b1b6a9005d56fa8b7666d39f934e48b2b"} Oct 31 19:39:09.971026 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.970788 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:39:09.976859 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.973682 4910 generic.go:356] "Generic (PLEG): container finished" podID="336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52" containerID="9c01437dae8454927a1d8cf486034b36fbb6999e0bd0d8a8793e664be8078ca3" exitCode=0 Oct 31 19:39:09.976859 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.973785 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" event={"ID":"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52","Type":"ContainerDied","Data":"9c01437dae8454927a1d8cf486034b36fbb6999e0bd0d8a8793e664be8078ca3"} Oct 31 19:39:09.977726 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.977697 4910 generic.go:356] "Generic (PLEG): container finished" podID="09156e2c-03fe-4f93-af3a-4449f61f1e2c" containerID="1ca447bd12143f53b0b0c4b345c1fe9e80305c79c26614cb1d824b359454ec3a" exitCode=0 Oct 31 19:39:09.977952 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.977929 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" event={"ID":"09156e2c-03fe-4f93-af3a-4449f61f1e2c","Type":"ContainerDied","Data":"1ca447bd12143f53b0b0c4b345c1fe9e80305c79c26614cb1d824b359454ec3a"} Oct 31 19:39:09.981725 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.981700 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16ace94-jqddp" event={"ID":"0ebff398-c9da-4829-910b-497908ea7feb","Type":"ContainerDied","Data":"8b977f009e8f8b63ec3de968a62cbbe30542fa37528c7d2663a368cb9a1cb842"} Oct 31 19:39:09.981825 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.981746 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b977f009e8f8b63ec3de968a62cbbe30542fa37528c7d2663a368cb9a1cb842" Oct 31 19:39:09.981825 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.981807 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d16ace94-jqddp" Oct 31 19:39:09.991481 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.991446 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-data\") pod \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " Oct 31 19:39:09.991640 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.991559 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012ba66a-ce6f-4261-9516-e8f0aa8ff815-kube-api-access-p7wh5\" (UniqueName: \"kubernetes.io/projected/012ba66a-ce6f-4261-9516-e8f0aa8ff815-kube-api-access-p7wh5\") pod \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " Oct 31 19:39:09.991640 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.991594 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-credential-socket\") pod \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " Oct 31 19:39:09.991640 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.991629 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-workload-certs\") pod \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " Oct 31 19:39:09.991874 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.991669 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-podinfo\") pod \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " Oct 31 19:39:09.991874 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.991809 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istiod-ca-cert\") pod \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " Oct 31 19:39:09.991874 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.991842 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-envoy\") pod \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " Oct 31 19:39:09.992150 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.991878 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-token\" (UniqueName: \"kubernetes.io/projected/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-token\") pod \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " Oct 31 19:39:09.992150 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.991980 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-workload-socket\") pod \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\" (UID: \"012ba66a-ce6f-4261-9516-e8f0aa8ff815\") " Oct 31 19:39:09.992341 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.992260 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-workload-certs" pod "012ba66a-ce6f-4261-9516-e8f0aa8ff815" (UID: "012ba66a-ce6f-4261-9516-e8f0aa8ff815"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:09.992501 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.992439 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-data" pod "012ba66a-ce6f-4261-9516-e8f0aa8ff815" (UID: "012ba66a-ce6f-4261-9516-e8f0aa8ff815"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:09.993691 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.993648 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:09.993691 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.993676 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:09.994879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.994850 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istiod-ca-cert" pod "012ba66a-ce6f-4261-9516-e8f0aa8ff815" (UID: "012ba66a-ce6f-4261-9516-e8f0aa8ff815"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:39:09.995060 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.995040 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-credential-socket" pod "012ba66a-ce6f-4261-9516-e8f0aa8ff815" (UID: "012ba66a-ce6f-4261-9516-e8f0aa8ff815"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:09.995696 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:09.995675 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-workload-socket" pod "012ba66a-ce6f-4261-9516-e8f0aa8ff815" (UID: "012ba66a-ce6f-4261-9516-e8f0aa8ff815"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:10.010291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.009895 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-envoy" pod "012ba66a-ce6f-4261-9516-e8f0aa8ff815" (UID: "012ba66a-ce6f-4261-9516-e8f0aa8ff815"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:10.058111 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.035365 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-podinfo" pod "012ba66a-ce6f-4261-9516-e8f0aa8ff815" (UID: "012ba66a-ce6f-4261-9516-e8f0aa8ff815"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:39:10.058111 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.035502 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-token" pod "012ba66a-ce6f-4261-9516-e8f0aa8ff815" (UID: "012ba66a-ce6f-4261-9516-e8f0aa8ff815"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:39:10.076095 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.076055 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012ba66a-ce6f-4261-9516-e8f0aa8ff815-kube-api-access-p7wh5" pod "012ba66a-ce6f-4261-9516-e8f0aa8ff815" (UID: "012ba66a-ce6f-4261-9516-e8f0aa8ff815"). InnerVolumeSpecName "kube-api-access-p7wh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:39:10.098473 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.098437 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7wh5\" (UniqueName: \"kubernetes.io/projected/012ba66a-ce6f-4261-9516-e8f0aa8ff815-kube-api-access-p7wh5\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:10.099014 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.098982 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:10.099014 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.099014 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:10.099214 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.099027 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:10.099214 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.099040 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:10.099214 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.099054 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012ba66a-ce6f-4261-9516-e8f0aa8ff815-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:10.099214 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.099067 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012ba66a-ce6f-4261-9516-e8f0aa8ff815-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:10.131895 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.130893 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:39:10.204140 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.204101 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istiod-ca-cert\") pod \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " Oct 31 19:39:10.205976 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.204663 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istiod-ca-cert" pod "09156e2c-03fe-4f93-af3a-4449f61f1e2c" (UID: "09156e2c-03fe-4f93-af3a-4449f61f1e2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:39:10.206339 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.206303 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-credential-socket" pod "09156e2c-03fe-4f93-af3a-4449f61f1e2c" (UID: "09156e2c-03fe-4f93-af3a-4449f61f1e2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:10.206463 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.206450 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-credential-socket\") pod \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " Oct 31 19:39:10.206551 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.206538 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-token\" (UniqueName: \"kubernetes.io/projected/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-token\") pod \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " Oct 31 19:39:10.206995 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.206969 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-envoy\") pod \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " Oct 31 19:39:10.207176 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.207154 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09156e2c-03fe-4f93-af3a-4449f61f1e2c-kube-api-access-4dnj7\" (UniqueName: \"kubernetes.io/projected/09156e2c-03fe-4f93-af3a-4449f61f1e2c-kube-api-access-4dnj7\") pod \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " Oct 31 19:39:10.207274 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.207262 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-podinfo\") pod \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " Oct 31 19:39:10.207457 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.207439 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-data\") pod \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " Oct 31 19:39:10.207557 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.207547 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-workload-certs\") pod \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " Oct 31 19:39:10.207650 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.207641 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-workload-socket\") pod \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " Oct 31 19:39:10.210595 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.210221 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-data" pod "09156e2c-03fe-4f93-af3a-4449f61f1e2c" (UID: "09156e2c-03fe-4f93-af3a-4449f61f1e2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:10.210595 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.210391 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-workload-socket" pod "09156e2c-03fe-4f93-af3a-4449f61f1e2c" (UID: "09156e2c-03fe-4f93-af3a-4449f61f1e2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:10.210595 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.210579 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-workload-certs" pod "09156e2c-03fe-4f93-af3a-4449f61f1e2c" (UID: "09156e2c-03fe-4f93-af3a-4449f61f1e2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:10.213089 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.212695 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:39:10.215069 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.215046 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09156e2c-03fe-4f93-af3a-4449f61f1e2c-kube-api-access-4dnj7" pod "09156e2c-03fe-4f93-af3a-4449f61f1e2c" (UID: "09156e2c-03fe-4f93-af3a-4449f61f1e2c"). InnerVolumeSpecName "kube-api-access-4dnj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:39:10.303005 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.290535 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-envoy" pod "09156e2c-03fe-4f93-af3a-4449f61f1e2c" (UID: "09156e2c-03fe-4f93-af3a-4449f61f1e2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:10.382564 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.382526 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-podinfo" pod "09156e2c-03fe-4f93-af3a-4449f61f1e2c" (UID: "09156e2c-03fe-4f93-af3a-4449f61f1e2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:39:10.384156 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.384122 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-podinfo\") pod \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\" (UID: \"09156e2c-03fe-4f93-af3a-4449f61f1e2c\") " Oct 31 19:39:10.384303 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.384173 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-token" pod "09156e2c-03fe-4f93-af3a-4449f61f1e2c" (UID: "09156e2c-03fe-4f93-af3a-4449f61f1e2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:39:10.384303 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.384247 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-workload-socket\") pod \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " Oct 31 19:39:10.384417 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:39:10.384319 4910 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/09156e2c-03fe-4f93-af3a-4449f61f1e2c/volumes/kubernetes.io~downward-api/istio-podinfo Oct 31 19:39:10.384417 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.384336 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-podinfo" pod "09156e2c-03fe-4f93-af3a-4449f61f1e2c" (UID: "09156e2c-03fe-4f93-af3a-4449f61f1e2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:39:10.384612 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.384587 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-workload-socket" pod "336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52" (UID: "336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:10.385880 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.385858 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-credential-socket\") pod \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " Oct 31 19:39:10.386187 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.386154 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-credential-socket" pod "336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52" (UID: "336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:10.386483 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.386154 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-envoy\") pod \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " Oct 31 19:39:10.386573 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.386528 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-data\") pod \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " Oct 31 19:39:10.386628 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.386580 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-podinfo\") pod \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " Oct 31 19:39:10.386682 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.386647 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-kube-api-access-9h96v\" (UniqueName: \"kubernetes.io/projected/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-kube-api-access-9h96v\") pod \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " Oct 31 19:39:10.387072 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.386876 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istiod-ca-cert\") pod \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " Oct 31 19:39:10.387072 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.386956 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-workload-certs\") pod \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " Oct 31 19:39:10.387072 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.387007 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-token\" (UniqueName: \"kubernetes.io/projected/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-token\") pod \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\" (UID: \"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52\") " Oct 31 19:39:10.387822 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.387564 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istiod-ca-cert" pod "336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52" (UID: "336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:39:10.388213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.388177 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-workload-certs" pod "336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52" (UID: "336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:10.388326 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.388209 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:10.388326 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.388238 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:10.388326 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.388263 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:10.388326 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.388278 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:10.388326 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.388293 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:10.388326 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.388312 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:10.388645 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.388327 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:10.388645 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.388342 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:10.388645 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.388356 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:10.388645 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.388376 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:10.388645 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.388391 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dnj7\" (UniqueName: \"kubernetes.io/projected/09156e2c-03fe-4f93-af3a-4449f61f1e2c-kube-api-access-4dnj7\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:10.388645 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.388406 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09156e2c-03fe-4f93-af3a-4449f61f1e2c-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:10.389226 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.389199 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-data" pod "336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52" (UID: "336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:10.390211 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.390177 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-envoy" pod "336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52" (UID: "336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:10.395203 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.395174 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-token" pod "336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52" (UID: "336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:39:10.395375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.395350 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-kube-api-access-9h96v" pod "336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52" (UID: "336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52"). InnerVolumeSpecName "kube-api-access-9h96v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:39:10.395500 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.395485 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-podinfo" pod "336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52" (UID: "336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:39:10.489885 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.489848 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:10.489885 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.489883 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:10.489885 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.489893 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:10.490141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.489902 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:10.490141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.489926 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:10.490141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.489936 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h96v\" (UniqueName: \"kubernetes.io/projected/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52-kube-api-access-9h96v\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:10.782397 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.782303 4910 patch_prober.go:29] interesting pod/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 31 19:39:10.782653 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.782621 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" podUID="790bd78d-4525-4587-9b6f-30af5841a88a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 31 19:39:10.992812 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.992688 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" Oct 31 19:39:10.992812 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.992708 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf" event={"ID":"09156e2c-03fe-4f93-af3a-4449f61f1e2c","Type":"ContainerDied","Data":"87e3c7b7812e781f9c8b6cfa0621950e01a8f705df4f82cc9163ecddae5b8fb5"} Oct 31 19:39:10.992812 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.992784 4910 scope.go:122] "RemoveContainer" containerID="1ca447bd12143f53b0b0c4b345c1fe9e80305c79c26614cb1d824b359454ec3a" Oct 31 19:39:10.998497 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.994809 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" event={"ID":"012ba66a-ce6f-4261-9516-e8f0aa8ff815","Type":"ContainerDied","Data":"7d13ac87036da33f3e7f7aebd8617a6e4142029c52a76d0d0c59a27f891e0f98"} Oct 31 19:39:10.998497 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.995034 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9" Oct 31 19:39:10.998497 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.996487 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" event={"ID":"336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52","Type":"ContainerDied","Data":"bf5cd0c60e887a3fb7a55cc4dfb4466540132950b27da023e3866f9cfd367871"} Oct 31 19:39:10.998497 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:10.996598 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z" Oct 31 19:39:11.014518 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:11.014501 4910 scope.go:122] "RemoveContainer" containerID="f3dfb43990ecb35499512e9cf0b6372b1b6a9005d56fa8b7666d39f934e48b2b" Oct 31 19:39:11.062930 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:11.058101 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf"] Oct 31 19:39:11.066541 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:11.066515 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-c9btf"] Oct 31 19:39:11.068450 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:11.068428 4910 scope.go:122] "RemoveContainer" containerID="9c01437dae8454927a1d8cf486034b36fbb6999e0bd0d8a8793e664be8078ca3" Oct 31 19:39:11.097795 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:11.097768 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9"] Oct 31 19:39:11.109012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:11.108987 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-8crc9"] Oct 31 19:39:11.161115 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:11.161081 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z"] Oct 31 19:39:11.181014 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:11.180982 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-chr6z"] Oct 31 19:39:11.758334 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:11.758280 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="012ba66a-ce6f-4261-9516-e8f0aa8ff815" path="/var/lib/kubelet/pods/012ba66a-ce6f-4261-9516-e8f0aa8ff815/volumes" Oct 31 19:39:11.759065 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:11.759042 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09156e2c-03fe-4f93-af3a-4449f61f1e2c" path="/var/lib/kubelet/pods/09156e2c-03fe-4f93-af3a-4449f61f1e2c/volumes" Oct 31 19:39:11.759600 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:11.759579 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52" path="/var/lib/kubelet/pods/336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52/volumes" Oct 31 19:39:11.786676 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:11.786653 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:39:11.787079 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:11.787057 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:39:11.787698 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:11.787683 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:39:12.715834 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:39:12.715797 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:39:14.908132 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:14.908101 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-2q285"] Oct 31 19:39:14.969078 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:14.967228 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-2q285"] Oct 31 19:39:15.758224 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:15.758193 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99fa69c0-fb59-4711-9d25-2813eeaba33b" path="/var/lib/kubelet/pods/99fa69c0-fb59-4711-9d25-2813eeaba33b/volumes" Oct 31 19:39:15.766249 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:15.766220 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z"] Oct 31 19:39:15.766951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:15.766896 4910 cpu_manager.go:414] "containerMap: removing container" podUID="012ba66a-ce6f-4261-9516-e8f0aa8ff815" containerName="istio-proxy" Oct 31 19:39:15.766951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:15.766931 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="012ba66a-ce6f-4261-9516-e8f0aa8ff815" containerName="istio-proxy" Oct 31 19:39:15.767132 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:15.766957 4910 cpu_manager.go:414] "containerMap: removing container" podUID="09156e2c-03fe-4f93-af3a-4449f61f1e2c" containerName="istio-proxy" Oct 31 19:39:15.767132 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:15.766964 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09156e2c-03fe-4f93-af3a-4449f61f1e2c" containerName="istio-proxy" Oct 31 19:39:15.767132 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:15.766999 4910 cpu_manager.go:414] "containerMap: removing container" podUID="0ebff398-c9da-4829-910b-497908ea7feb" containerName="revoke-keys" Oct 31 19:39:15.767132 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:15.767006 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ebff398-c9da-4829-910b-497908ea7feb" containerName="revoke-keys" Oct 31 19:39:15.767132 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:15.767025 4910 cpu_manager.go:414] "containerMap: removing container" podUID="336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52" containerName="istio-proxy" Oct 31 19:39:15.767132 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:15.767032 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52" containerName="istio-proxy" Oct 31 19:39:15.767414 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:15.767203 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ebff398-c9da-4829-910b-497908ea7feb" containerName="revoke-keys" Oct 31 19:39:15.767414 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:15.767219 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="336cb7b6-2b37-4e93-9c0c-1a2c07e8ca52" containerName="istio-proxy" Oct 31 19:39:15.767414 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:15.767240 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="012ba66a-ce6f-4261-9516-e8f0aa8ff815" containerName="istio-proxy" Oct 31 19:39:15.767414 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:15.767254 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="09156e2c-03fe-4f93-af3a-4449f61f1e2c" containerName="istio-proxy" Oct 31 19:39:15.771389 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:15.771366 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:39:15.784513 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:15.784486 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z"] Oct 31 19:39:15.917105 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:15.917053 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7031809a-4dff-4444-bd5c-521501265431-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l4p8z\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:39:15.917511 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:15.917226 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7031809a-4dff-4444-bd5c-521501265431-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l4p8z\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:39:15.917511 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:15.917426 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7031809a-4dff-4444-bd5c-521501265431-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l4p8z\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:39:15.917589 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:15.917556 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tctf9\" (UniqueName: \"kubernetes.io/projected/7031809a-4dff-4444-bd5c-521501265431-kube-api-access-tctf9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l4p8z\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:39:15.917623 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:15.917602 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7031809a-4dff-4444-bd5c-521501265431-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l4p8z\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:39:15.917751 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:15.917730 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7031809a-4dff-4444-bd5c-521501265431-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l4p8z\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:39:15.917842 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:15.917824 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7031809a-4dff-4444-bd5c-521501265431-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l4p8z\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:39:16.019203 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:16.019109 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7031809a-4dff-4444-bd5c-521501265431-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l4p8z\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:39:16.019358 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:16.019217 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7031809a-4dff-4444-bd5c-521501265431-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l4p8z\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:39:16.019405 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:16.019353 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7031809a-4dff-4444-bd5c-521501265431-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l4p8z\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:39:16.019452 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:16.019434 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tctf9\" (UniqueName: \"kubernetes.io/projected/7031809a-4dff-4444-bd5c-521501265431-kube-api-access-tctf9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l4p8z\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:39:16.019497 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:16.019476 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7031809a-4dff-4444-bd5c-521501265431-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l4p8z\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:39:16.019555 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:16.019540 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7031809a-4dff-4444-bd5c-521501265431-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l4p8z\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:39:16.019633 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:16.019619 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7031809a-4dff-4444-bd5c-521501265431-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l4p8z\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:39:16.022771 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:16.022744 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7031809a-4dff-4444-bd5c-521501265431-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l4p8z\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:39:16.024433 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:16.024395 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7031809a-4dff-4444-bd5c-521501265431-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l4p8z\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:39:16.024620 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:16.024486 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7031809a-4dff-4444-bd5c-521501265431-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l4p8z\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:39:16.025619 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:16.025595 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7031809a-4dff-4444-bd5c-521501265431-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l4p8z\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:39:16.026963 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:16.026881 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7031809a-4dff-4444-bd5c-521501265431-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l4p8z\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:39:16.033595 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:16.033573 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d508d5-jk8c5"] Oct 31 19:39:16.034352 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:16.034328 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7031809a-4dff-4444-bd5c-521501265431-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l4p8z\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:39:16.038612 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:16.038592 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tctf9\" (UniqueName: \"kubernetes.io/projected/7031809a-4dff-4444-bd5c-521501265431-kube-api-access-tctf9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l4p8z\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:39:16.041517 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:16.041495 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d508d5-jk8c5"] Oct 31 19:39:16.090706 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:16.090675 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:39:16.391885 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:39:16.391856 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7031809a_4dff_4444_bd5c_521501265431.slice/crio-00755ba6e121432771c8a1c63719f0f44f4100df72eb1e483cf795a8c76b448b WatchSource:0}: Error finding container 00755ba6e121432771c8a1c63719f0f44f4100df72eb1e483cf795a8c76b448b: Status 404 returned error can't find the container with id 00755ba6e121432771c8a1c63719f0f44f4100df72eb1e483cf795a8c76b448b Oct 31 19:39:16.395360 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:16.395318 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:39:16.395461 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:16.395386 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:39:16.395823 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:16.395790 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z"] Oct 31 19:39:17.166179 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:17.166141 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" event={"ID":"7031809a-4dff-4444-bd5c-521501265431","Type":"ContainerStarted","Data":"c59dd42a41180198b2e8d88811ce2a90bc69574b8562de2df31275bd96dafd31"} Oct 31 19:39:17.166740 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:17.166202 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" event={"ID":"7031809a-4dff-4444-bd5c-521501265431","Type":"ContainerStarted","Data":"00755ba6e121432771c8a1c63719f0f44f4100df72eb1e483cf795a8c76b448b"} Oct 31 19:39:17.167019 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:17.166998 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:39:17.168737 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:17.168679 4910 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-l4p8z container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:39:17.168839 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:17.168763 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" podUID="7031809a-4dff-4444-bd5c-521501265431" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:39:17.208557 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:17.208488 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" podStartSLOduration=2.208467863 podStartE2EDuration="2.208467863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:17.201470498 +0000 UTC m=+1191.564290871" watchObservedRunningTime="2026-10-31 19:39:17.208467863 +0000 UTC m=+1191.571288237" Oct 31 19:39:17.772752 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:17.772710 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faeecdd8-67fc-4249-ba29-ab06d605de57" path="/var/lib/kubelet/pods/faeecdd8-67fc-4249-ba29-ab06d605de57/volumes" Oct 31 19:39:18.175994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:18.175964 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:39:22.073429 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:22.073382 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e847aa56-zzsnz"] Oct 31 19:39:22.078877 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:22.078799 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aa3d18b-2tr54"] Oct 31 19:39:22.083756 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:22.083726 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e847aa56-zzsnz"] Oct 31 19:39:22.087831 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:22.087803 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aa3d18b-2tr54"] Oct 31 19:39:23.757919 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:23.757833 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfdfc824-4fbb-4053-ada5-1ab32efa036b" path="/var/lib/kubelet/pods/dfdfc824-4fbb-4053-ada5-1ab32efa036b/volumes" Oct 31 19:39:23.758402 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:23.758387 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4b24c82-9dba-4b45-994c-dbe26498219d" path="/var/lib/kubelet/pods/e4b24c82-9dba-4b45-994c-dbe26498219d/volumes" Oct 31 19:39:26.105616 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:26.105575 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 19:39:26.116033 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:26.116011 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 19:39:27.492540 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:27.492502 4910 scope.go:122] "RemoveContainer" containerID="63064eddc5527ba3eda0deaaf2e3e5f6f678588a2f33e0f608fd7e03da58147f" Oct 31 19:39:27.505823 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:27.505801 4910 scope.go:122] "RemoveContainer" containerID="06280f91b511ede7a2aa6ee10c435d508f0a262c36c757001eb03460e7524e7e" Oct 31 19:39:27.516323 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:27.516291 4910 scope.go:122] "RemoveContainer" containerID="a0660493712715953240af4379666ebd0774d303541ea8bd2f868bc1ddf78f5e" Oct 31 19:39:27.529538 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:27.529516 4910 scope.go:122] "RemoveContainer" containerID="e56d6038808de5de60e3f3681138d46d18a04f5efe56133aa0a05ef04a226f05" Oct 31 19:39:27.540887 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:27.540867 4910 scope.go:122] "RemoveContainer" containerID="cd4d7f26c0e8d7e8b79e6da4fcf702f4de112c483efead0c0463e063413fcf33" Oct 31 19:39:29.046030 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:29.045991 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abe32a79-lgvdb"] Oct 31 19:39:29.052212 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:29.052184 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abe32a79-lgvdb"] Oct 31 19:39:29.758828 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:29.758790 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebb4ece9-4a96-4027-969a-6a4884f5205b" path="/var/lib/kubelet/pods/ebb4ece9-4a96-4027-969a-6a4884f5205b/volumes" Oct 31 19:39:34.654153 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:34.645140 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v"] Oct 31 19:39:34.677946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:34.667378 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" podUID="0a24bfa8-34c0-4825-876a-c2fec6019829" containerName="istio-proxy" containerID="cri-o://28664d6bcfb9b6b8214d9dd2d1320f7bd7de6dd2f021a361d2db9d1085eab3e1" gracePeriod=30 Oct 31 19:39:35.090076 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:35.090033 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:39:35.097538 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:35.090604 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:39:35.204084 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:35.125500 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:39:35.226948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:35.215597 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:39:35.260778 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:35.260731 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:39:35.311936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:35.307999 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:39:36.057955 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:36.055061 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:39:36.251499 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:36.251452 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:39:36.252361 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:36.252325 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:39:36.262977 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:36.262296 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:39:36.443583 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:36.443502 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:39:36.511426 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:36.508360 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:39:40.116719 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.116690 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:39:40.193700 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.193662 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a24bfa8-34c0-4825-876a-c2fec6019829-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a24bfa8-34c0-4825-876a-c2fec6019829-istiod-ca-cert\") pod \"0a24bfa8-34c0-4825-876a-c2fec6019829\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " Oct 31 19:39:40.193882 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.193734 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-workload-socket\") pod \"0a24bfa8-34c0-4825-876a-c2fec6019829\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " Oct 31 19:39:40.193882 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.193852 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-workload-certs\") pod \"0a24bfa8-34c0-4825-876a-c2fec6019829\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " Oct 31 19:39:40.194037 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.194017 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-podinfo\") pod \"0a24bfa8-34c0-4825-876a-c2fec6019829\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " Oct 31 19:39:40.194110 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.194069 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-envoy\") pod \"0a24bfa8-34c0-4825-876a-c2fec6019829\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " Oct 31 19:39:40.194169 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.194094 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a24bfa8-34c0-4825-876a-c2fec6019829-istiod-ca-cert" pod "0a24bfa8-34c0-4825-876a-c2fec6019829" (UID: "0a24bfa8-34c0-4825-876a-c2fec6019829"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:39:40.194169 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.194147 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-token\" (UniqueName: \"kubernetes.io/projected/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-token\") pod \"0a24bfa8-34c0-4825-876a-c2fec6019829\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " Oct 31 19:39:40.194275 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.194228 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-data\") pod \"0a24bfa8-34c0-4825-876a-c2fec6019829\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " Oct 31 19:39:40.194374 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.194359 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a24bfa8-34c0-4825-876a-c2fec6019829-kube-api-access-mtq2d\" (UniqueName: \"kubernetes.io/projected/0a24bfa8-34c0-4825-876a-c2fec6019829-kube-api-access-mtq2d\") pod \"0a24bfa8-34c0-4825-876a-c2fec6019829\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " Oct 31 19:39:40.194447 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.194400 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-workload-certs" pod "0a24bfa8-34c0-4825-876a-c2fec6019829" (UID: "0a24bfa8-34c0-4825-876a-c2fec6019829"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:40.194447 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.194416 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-credential-socket\") pod \"0a24bfa8-34c0-4825-876a-c2fec6019829\" (UID: \"0a24bfa8-34c0-4825-876a-c2fec6019829\") " Oct 31 19:39:40.194601 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.194580 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-workload-socket" pod "0a24bfa8-34c0-4825-876a-c2fec6019829" (UID: "0a24bfa8-34c0-4825-876a-c2fec6019829"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:40.195127 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.195106 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a24bfa8-34c0-4825-876a-c2fec6019829-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:40.195295 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.195132 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:40.195295 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.195146 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:40.195450 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.195360 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-credential-socket" pod "0a24bfa8-34c0-4825-876a-c2fec6019829" (UID: "0a24bfa8-34c0-4825-876a-c2fec6019829"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:40.195949 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.195889 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-data" pod "0a24bfa8-34c0-4825-876a-c2fec6019829" (UID: "0a24bfa8-34c0-4825-876a-c2fec6019829"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:40.198007 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.197979 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-token" pod "0a24bfa8-34c0-4825-876a-c2fec6019829" (UID: "0a24bfa8-34c0-4825-876a-c2fec6019829"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:39:40.199050 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.199023 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-podinfo" pod "0a24bfa8-34c0-4825-876a-c2fec6019829" (UID: "0a24bfa8-34c0-4825-876a-c2fec6019829"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:39:40.205890 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.205865 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-envoy" pod "0a24bfa8-34c0-4825-876a-c2fec6019829" (UID: "0a24bfa8-34c0-4825-876a-c2fec6019829"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:40.207446 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.207422 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a24bfa8-34c0-4825-876a-c2fec6019829-kube-api-access-mtq2d" pod "0a24bfa8-34c0-4825-876a-c2fec6019829" (UID: "0a24bfa8-34c0-4825-876a-c2fec6019829"). InnerVolumeSpecName "kube-api-access-mtq2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:39:40.296764 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.296722 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:40.296764 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.296762 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:40.297014 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.296777 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:40.297014 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.296791 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:40.297014 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.296806 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtq2d\" (UniqueName: \"kubernetes.io/projected/0a24bfa8-34c0-4825-876a-c2fec6019829-kube-api-access-mtq2d\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:40.297014 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.296821 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a24bfa8-34c0-4825-876a-c2fec6019829-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:40.408581 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.408547 4910 generic.go:356] "Generic (PLEG): container finished" podID="0a24bfa8-34c0-4825-876a-c2fec6019829" containerID="28664d6bcfb9b6b8214d9dd2d1320f7bd7de6dd2f021a361d2db9d1085eab3e1" exitCode=0 Oct 31 19:39:40.408737 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.408620 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" Oct 31 19:39:40.408737 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.408628 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" event={"ID":"0a24bfa8-34c0-4825-876a-c2fec6019829","Type":"ContainerDied","Data":"28664d6bcfb9b6b8214d9dd2d1320f7bd7de6dd2f021a361d2db9d1085eab3e1"} Oct 31 19:39:40.408737 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.408683 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v" event={"ID":"0a24bfa8-34c0-4825-876a-c2fec6019829","Type":"ContainerDied","Data":"9ae8891146d4c577883c034964c177ff661d0691c96f323176465511c986d589"} Oct 31 19:39:40.408737 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.408705 4910 scope.go:122] "RemoveContainer" containerID="28664d6bcfb9b6b8214d9dd2d1320f7bd7de6dd2f021a361d2db9d1085eab3e1" Oct 31 19:39:40.421902 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.421884 4910 scope.go:122] "RemoveContainer" containerID="28664d6bcfb9b6b8214d9dd2d1320f7bd7de6dd2f021a361d2db9d1085eab3e1" Oct 31 19:39:40.422260 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:39:40.422239 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28664d6bcfb9b6b8214d9dd2d1320f7bd7de6dd2f021a361d2db9d1085eab3e1\": container with ID starting with 28664d6bcfb9b6b8214d9dd2d1320f7bd7de6dd2f021a361d2db9d1085eab3e1 not found: ID does not exist" containerID="28664d6bcfb9b6b8214d9dd2d1320f7bd7de6dd2f021a361d2db9d1085eab3e1" Oct 31 19:39:40.422353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.422272 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28664d6bcfb9b6b8214d9dd2d1320f7bd7de6dd2f021a361d2db9d1085eab3e1"} err="failed to get container status \"28664d6bcfb9b6b8214d9dd2d1320f7bd7de6dd2f021a361d2db9d1085eab3e1\": rpc error: code = NotFound desc = could not find container \"28664d6bcfb9b6b8214d9dd2d1320f7bd7de6dd2f021a361d2db9d1085eab3e1\": container with ID starting with 28664d6bcfb9b6b8214d9dd2d1320f7bd7de6dd2f021a361d2db9d1085eab3e1 not found: ID does not exist" Oct 31 19:39:40.446469 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.446439 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v"] Oct 31 19:39:40.454283 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:40.454246 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-cr28v"] Oct 31 19:39:41.767254 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:41.766712 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a24bfa8-34c0-4825-876a-c2fec6019829" path="/var/lib/kubelet/pods/0a24bfa8-34c0-4825-876a-c2fec6019829/volumes" Oct 31 19:39:42.718362 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:39:42.718325 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:39:43.036407 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:43.036309 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de1f4432-5xcqs"] Oct 31 19:39:43.040796 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:43.040767 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de1f4432-5xcqs"] Oct 31 19:39:43.770114 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:43.770070 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc81cb01-6336-47e0-8d44-466ccaa83704" path="/var/lib/kubelet/pods/bc81cb01-6336-47e0-8d44-466ccaa83704/volumes" Oct 31 19:39:44.509373 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:44.509292 4910 generic.go:356] "Generic (PLEG): container finished" podID="922b063f-4aee-4ed2-9355-3cd5222862df" containerID="f40141a0f0edd3d9dbf89b3b2852783ef462ab01f27fe56d3f41f5db6e47b93a" exitCode=0 Oct 31 19:39:44.509373 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:44.509346 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" event={"ID":"922b063f-4aee-4ed2-9355-3cd5222862df","Type":"ContainerDied","Data":"f40141a0f0edd3d9dbf89b3b2852783ef462ab01f27fe56d3f41f5db6e47b93a"} Oct 31 19:39:44.509988 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:44.509962 4910 scope.go:122] "RemoveContainer" containerID="f40141a0f0edd3d9dbf89b3b2852783ef462ab01f27fe56d3f41f5db6e47b93a" Oct 31 19:39:44.512360 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:44.512321 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:39:44.512605 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:44.512581 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:39:44.512880 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:44.512851 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:39:45.555585 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:45.555540 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" event={"ID":"922b063f-4aee-4ed2-9355-3cd5222862df","Type":"ContainerStarted","Data":"ea1885212f37c461eb2ee862b79f1b42373c21f2cf365a087849a56c55ac8550"} Oct 31 19:39:45.878004 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:45.877965 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:39:45.878259 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:45.878236 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 19:39:45.878315 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:45.878292 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" podUID="922b063f-4aee-4ed2-9355-3cd5222862df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 19:39:46.688866 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:46.688831 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z"] Oct 31 19:39:46.689507 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:46.689493 4910 cpu_manager.go:414] "containerMap: removing container" podUID="0a24bfa8-34c0-4825-876a-c2fec6019829" containerName="istio-proxy" Oct 31 19:39:46.689557 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:46.689507 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a24bfa8-34c0-4825-876a-c2fec6019829" containerName="istio-proxy" Oct 31 19:39:46.689657 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:46.689648 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a24bfa8-34c0-4825-876a-c2fec6019829" containerName="istio-proxy" Oct 31 19:39:46.693564 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:46.693544 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z" Oct 31 19:39:46.704352 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:46.704321 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z"] Oct 31 19:39:46.704740 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:46.704720 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:39:46.705150 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:46.705017 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:39:46.784103 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:46.784058 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc7e604e-aa70-438f-9fbc-ee824a47732d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z\" (UID: \"cc7e604e-aa70-438f-9fbc-ee824a47732d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z" Oct 31 19:39:46.784326 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:46.784194 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc7e604e-aa70-438f-9fbc-ee824a47732d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z\" (UID: \"cc7e604e-aa70-438f-9fbc-ee824a47732d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z" Oct 31 19:39:46.784326 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:46.784270 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnv6t\" (UniqueName: \"kubernetes.io/projected/cc7e604e-aa70-438f-9fbc-ee824a47732d-kube-api-access-xnv6t\") pod \"maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z\" (UID: \"cc7e604e-aa70-438f-9fbc-ee824a47732d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z" Oct 31 19:39:46.877671 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:46.877632 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 19:39:46.877840 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:46.877688 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" podUID="922b063f-4aee-4ed2-9355-3cd5222862df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 19:39:46.885592 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:46.885559 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc7e604e-aa70-438f-9fbc-ee824a47732d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z\" (UID: \"cc7e604e-aa70-438f-9fbc-ee824a47732d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z" Oct 31 19:39:46.885726 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:46.885656 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnv6t\" (UniqueName: \"kubernetes.io/projected/cc7e604e-aa70-438f-9fbc-ee824a47732d-kube-api-access-xnv6t\") pod \"maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z\" (UID: \"cc7e604e-aa70-438f-9fbc-ee824a47732d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z" Oct 31 19:39:46.886150 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:46.886123 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc7e604e-aa70-438f-9fbc-ee824a47732d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z\" (UID: \"cc7e604e-aa70-438f-9fbc-ee824a47732d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z" Oct 31 19:39:46.886738 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:46.886713 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc7e604e-aa70-438f-9fbc-ee824a47732d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z\" (UID: \"cc7e604e-aa70-438f-9fbc-ee824a47732d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z" Oct 31 19:39:46.888045 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:46.888021 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc7e604e-aa70-438f-9fbc-ee824a47732d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z\" (UID: \"cc7e604e-aa70-438f-9fbc-ee824a47732d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z" Oct 31 19:39:46.901381 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:46.901351 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnv6t\" (UniqueName: \"kubernetes.io/projected/cc7e604e-aa70-438f-9fbc-ee824a47732d-kube-api-access-xnv6t\") pod \"maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z\" (UID: \"cc7e604e-aa70-438f-9fbc-ee824a47732d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z" Oct 31 19:39:47.017793 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:47.017714 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z" Oct 31 19:39:47.601568 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:39:47.601537 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7e604e_aa70_438f_9fbc_ee824a47732d.slice/crio-b551b428c9c14ef8f0f59ff9028b8aa8561827049935c415b11eeb1b3aae8060 WatchSource:0}: Error finding container b551b428c9c14ef8f0f59ff9028b8aa8561827049935c415b11eeb1b3aae8060: Status 404 returned error can't find the container with id b551b428c9c14ef8f0f59ff9028b8aa8561827049935c415b11eeb1b3aae8060 Oct 31 19:39:47.603996 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:47.603966 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z"] Oct 31 19:39:47.878375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:47.878343 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 19:39:47.878750 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:47.878396 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" podUID="922b063f-4aee-4ed2-9355-3cd5222862df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 19:39:48.588944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:48.587194 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z" event={"ID":"cc7e604e-aa70-438f-9fbc-ee824a47732d","Type":"ContainerStarted","Data":"41e7fb032dface71aa0ba3931c99c036f6a79fe5f6a11832789bc46e8200faf4"} Oct 31 19:39:48.588944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:48.587251 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z" event={"ID":"cc7e604e-aa70-438f-9fbc-ee824a47732d","Type":"ContainerStarted","Data":"b551b428c9c14ef8f0f59ff9028b8aa8561827049935c415b11eeb1b3aae8060"} Oct 31 19:39:48.614607 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:48.614534 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z" podStartSLOduration=2.614515754 podStartE2EDuration="2.614515754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:48.607698782 +0000 UTC m=+1222.970519151" watchObservedRunningTime="2026-10-31 19:39:48.614515754 +0000 UTC m=+1222.977336127" Oct 31 19:39:48.693701 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:48.693667 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk"] Oct 31 19:39:48.716503 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:48.713971 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk"] Oct 31 19:39:48.716503 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:48.714096 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:48.734483 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:48.734451 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e73256-openshift-default-dockercfg-dp7zl\"" Oct 31 19:39:48.836047 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:48.831417 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psjfm\" (UniqueName: \"kubernetes.io/projected/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-kube-api-access-psjfm\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:48.836047 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:48.831673 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-envoy\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:48.836047 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:48.831753 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-workload-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:48.836047 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:48.831792 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-workload-certs\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:48.836047 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:48.831827 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-data\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:48.836047 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:48.831964 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istiod-ca-cert\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:48.836047 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:48.831997 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-credential-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:48.836047 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:48.832048 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-podinfo\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:48.836047 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:48.832146 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-token\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:48.889869 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:48.881521 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 19:39:48.889869 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:48.881580 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" podUID="922b063f-4aee-4ed2-9355-3cd5222862df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 19:39:48.946667 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:48.941791 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-workload-certs\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:48.946667 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:48.941847 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-data\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:48.971936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:48.956479 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-workload-certs\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:49.063983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:49.063331 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-data\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:49.085938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:49.064798 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istiod-ca-cert\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:49.085938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:49.064842 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-credential-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:49.085938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:49.064881 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-podinfo\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:49.085938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:49.064960 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-token\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:49.085938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:49.071305 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-credential-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:49.085938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:49.072003 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istiod-ca-cert\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:49.155018 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:49.153986 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psjfm\" (UniqueName: \"kubernetes.io/projected/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-kube-api-access-psjfm\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:49.183764 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:49.166098 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-podinfo\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:49.183764 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:49.172128 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-envoy\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:49.183764 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:49.172235 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-workload-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:49.183764 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:49.172810 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-workload-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:49.255932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:49.253494 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-token\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:49.343628 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:49.310989 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psjfm\" (UniqueName: \"kubernetes.io/projected/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-kube-api-access-psjfm\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:49.382940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:49.375015 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-envoy\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:49.565137 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:49.565092 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:49.603785 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:49.603749 4910 generic.go:356] "Generic (PLEG): container finished" podID="cc7e604e-aa70-438f-9fbc-ee824a47732d" containerID="41e7fb032dface71aa0ba3931c99c036f6a79fe5f6a11832789bc46e8200faf4" exitCode=0 Oct 31 19:39:49.603989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:49.603841 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z" event={"ID":"cc7e604e-aa70-438f-9fbc-ee824a47732d","Type":"ContainerDied","Data":"41e7fb032dface71aa0ba3931c99c036f6a79fe5f6a11832789bc46e8200faf4"} Oct 31 19:39:49.687341 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:49.686602 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt"] Oct 31 19:39:49.687341 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:49.687258 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" podUID="0ba0cb41-a7be-462c-9e92-7dccedd1d92f" containerName="istio-proxy" containerID="cri-o://e707ec070da09d75ca4ecc29b61cb4444cd40b7858c4dc21e9cc5968d43662be" gracePeriod=30 Oct 31 19:39:49.894338 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:49.894247 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 19:39:49.894963 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:49.894924 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" podUID="922b063f-4aee-4ed2-9355-3cd5222862df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 19:39:50.004974 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:39:50.002837 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507b2fb7_a22a_4cac_9d4a_23199f7fcbff.slice/crio-0b7899e610a1cc14c161aad710120d655869fd1bcbb22f59cc326584f1e9c696 WatchSource:0}: Error finding container 0b7899e610a1cc14c161aad710120d655869fd1bcbb22f59cc326584f1e9c696: Status 404 returned error can't find the container with id 0b7899e610a1cc14c161aad710120d655869fd1bcbb22f59cc326584f1e9c696 Oct 31 19:39:50.008710 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:50.008643 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:39:50.008839 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:50.008737 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:39:50.008839 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:50.008804 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:39:50.014144 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:50.014117 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk"] Oct 31 19:39:50.665935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:50.665299 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" event={"ID":"507b2fb7-a22a-4cac-9d4a-23199f7fcbff","Type":"ContainerStarted","Data":"8ecb05670fd8cd8251e58cd0e3bdb1a8db6549d99b0a85c8015e2a152950365c"} Oct 31 19:39:50.665935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:50.665364 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" event={"ID":"507b2fb7-a22a-4cac-9d4a-23199f7fcbff","Type":"ContainerStarted","Data":"0b7899e610a1cc14c161aad710120d655869fd1bcbb22f59cc326584f1e9c696"} Oct 31 19:39:50.705994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:50.702277 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" podStartSLOduration=2.702255557 podStartE2EDuration="2.702255557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:50.696536516 +0000 UTC m=+1225.059356888" watchObservedRunningTime="2026-10-31 19:39:50.702255557 +0000 UTC m=+1225.065075929" Oct 31 19:39:50.877450 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:50.877415 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 31 19:39:50.878355 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:50.878318 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" podUID="922b063f-4aee-4ed2-9355-3cd5222862df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 31 19:39:51.196147 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:51.196116 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z" Oct 31 19:39:51.261433 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:51.261400 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc7e604e-aa70-438f-9fbc-ee824a47732d-kube-api-access-xnv6t\" (UniqueName: \"kubernetes.io/projected/cc7e604e-aa70-438f-9fbc-ee824a47732d-kube-api-access-xnv6t\") pod \"cc7e604e-aa70-438f-9fbc-ee824a47732d\" (UID: \"cc7e604e-aa70-438f-9fbc-ee824a47732d\") " Oct 31 19:39:51.261645 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:51.261626 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc7e604e-aa70-438f-9fbc-ee824a47732d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc7e604e-aa70-438f-9fbc-ee824a47732d-maas-api-openshift-ca\") pod \"cc7e604e-aa70-438f-9fbc-ee824a47732d\" (UID: \"cc7e604e-aa70-438f-9fbc-ee824a47732d\") " Oct 31 19:39:51.261724 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:51.261695 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc7e604e-aa70-438f-9fbc-ee824a47732d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc7e604e-aa70-438f-9fbc-ee824a47732d-maas-api-xks-ca\") pod \"cc7e604e-aa70-438f-9fbc-ee824a47732d\" (UID: \"cc7e604e-aa70-438f-9fbc-ee824a47732d\") " Oct 31 19:39:51.262543 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:51.262516 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc7e604e-aa70-438f-9fbc-ee824a47732d-maas-api-openshift-ca" pod "cc7e604e-aa70-438f-9fbc-ee824a47732d" (UID: "cc7e604e-aa70-438f-9fbc-ee824a47732d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:39:51.277580 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:51.277547 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc7e604e-aa70-438f-9fbc-ee824a47732d-maas-api-xks-ca" pod "cc7e604e-aa70-438f-9fbc-ee824a47732d" (UID: "cc7e604e-aa70-438f-9fbc-ee824a47732d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:39:51.279654 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:51.279623 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc7e604e-aa70-438f-9fbc-ee824a47732d-kube-api-access-xnv6t" pod "cc7e604e-aa70-438f-9fbc-ee824a47732d" (UID: "cc7e604e-aa70-438f-9fbc-ee824a47732d"). InnerVolumeSpecName "kube-api-access-xnv6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:39:51.364456 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:51.364413 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc7e604e-aa70-438f-9fbc-ee824a47732d-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:51.364456 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:51.364444 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc7e604e-aa70-438f-9fbc-ee824a47732d-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:51.364456 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:51.364456 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnv6t\" (UniqueName: \"kubernetes.io/projected/cc7e604e-aa70-438f-9fbc-ee824a47732d-kube-api-access-xnv6t\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:51.569539 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:51.569399 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:51.583432 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:51.583401 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:51.685483 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:51.685455 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z" Oct 31 19:39:51.685675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:51.685479 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z" event={"ID":"cc7e604e-aa70-438f-9fbc-ee824a47732d","Type":"ContainerDied","Data":"b551b428c9c14ef8f0f59ff9028b8aa8561827049935c415b11eeb1b3aae8060"} Oct 31 19:39:51.685675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:51.685560 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b551b428c9c14ef8f0f59ff9028b8aa8561827049935c415b11eeb1b3aae8060" Oct 31 19:39:51.685816 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:51.685796 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:51.687761 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:51.687738 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:39:51.881870 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:51.881799 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:39:51.882329 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:51.882306 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:39:51.883190 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:51.883172 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 19:39:53.043222 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:53.043189 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30a66197-cn86c"] Oct 31 19:39:53.052532 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:53.052505 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30a66197-cn86c"] Oct 31 19:39:53.759051 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:53.759013 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="121ea68a-20ce-4362-b833-778115f1a9d8" path="/var/lib/kubelet/pods/121ea68a-20ce-4362-b833-778115f1a9d8/volumes" Oct 31 19:39:53.793260 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:53.793221 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:39:53.793426 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:53.793288 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" podUID="0ba0cb41-a7be-462c-9e92-7dccedd1d92f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:39:55.067656 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.067622 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:39:55.220580 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.220537 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-workload-certs\") pod \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " Oct 31 19:39:55.220865 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.220841 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istiod-ca-cert\") pod \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " Oct 31 19:39:55.221041 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.221015 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-kube-api-access-kwzvc\" (UniqueName: \"kubernetes.io/projected/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-kube-api-access-kwzvc\") pod \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " Oct 31 19:39:55.221225 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.221081 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-workload-socket\") pod \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " Oct 31 19:39:55.221345 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.221248 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-credential-socket\") pod \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " Oct 31 19:39:55.221457 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.221439 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-token\" (UniqueName: \"kubernetes.io/projected/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-token\") pod \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " Oct 31 19:39:55.221551 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.221503 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istiod-ca-cert" pod "0ba0cb41-a7be-462c-9e92-7dccedd1d92f" (UID: "0ba0cb41-a7be-462c-9e92-7dccedd1d92f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:39:55.221551 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.221520 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-data\") pod \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " Oct 31 19:39:55.221625 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.221592 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-envoy\") pod \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " Oct 31 19:39:55.221704 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.221646 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-podinfo\") pod \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\" (UID: \"0ba0cb41-a7be-462c-9e92-7dccedd1d92f\") " Oct 31 19:39:55.223451 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.223307 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:55.223451 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.223419 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-credential-socket" pod "0ba0cb41-a7be-462c-9e92-7dccedd1d92f" (UID: "0ba0cb41-a7be-462c-9e92-7dccedd1d92f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:55.226542 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.226460 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-workload-socket" pod "0ba0cb41-a7be-462c-9e92-7dccedd1d92f" (UID: "0ba0cb41-a7be-462c-9e92-7dccedd1d92f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:55.226542 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.226476 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-workload-certs" pod "0ba0cb41-a7be-462c-9e92-7dccedd1d92f" (UID: "0ba0cb41-a7be-462c-9e92-7dccedd1d92f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:55.226936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.226896 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-data" pod "0ba0cb41-a7be-462c-9e92-7dccedd1d92f" (UID: "0ba0cb41-a7be-462c-9e92-7dccedd1d92f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:55.230148 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.230089 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-envoy" pod "0ba0cb41-a7be-462c-9e92-7dccedd1d92f" (UID: "0ba0cb41-a7be-462c-9e92-7dccedd1d92f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:39:55.237267 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.237231 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-token" pod "0ba0cb41-a7be-462c-9e92-7dccedd1d92f" (UID: "0ba0cb41-a7be-462c-9e92-7dccedd1d92f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:39:55.241386 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.240672 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-kube-api-access-kwzvc" pod "0ba0cb41-a7be-462c-9e92-7dccedd1d92f" (UID: "0ba0cb41-a7be-462c-9e92-7dccedd1d92f"). InnerVolumeSpecName "kube-api-access-kwzvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:39:55.241386 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.240796 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-podinfo" pod "0ba0cb41-a7be-462c-9e92-7dccedd1d92f" (UID: "0ba0cb41-a7be-462c-9e92-7dccedd1d92f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:39:55.325159 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.325118 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:55.325159 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.325162 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwzvc\" (UniqueName: \"kubernetes.io/projected/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-kube-api-access-kwzvc\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:55.325414 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.325180 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:55.325414 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.325195 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:55.325414 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.325209 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:55.325414 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.325222 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:55.325414 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.325237 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:55.325414 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.325252 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ba0cb41-a7be-462c-9e92-7dccedd1d92f-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:39:55.833084 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.833050 4910 generic.go:356] "Generic (PLEG): container finished" podID="0ba0cb41-a7be-462c-9e92-7dccedd1d92f" containerID="e707ec070da09d75ca4ecc29b61cb4444cd40b7858c4dc21e9cc5968d43662be" exitCode=0 Oct 31 19:39:55.833260 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.833105 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" Oct 31 19:39:55.833260 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.833135 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" event={"ID":"0ba0cb41-a7be-462c-9e92-7dccedd1d92f","Type":"ContainerDied","Data":"e707ec070da09d75ca4ecc29b61cb4444cd40b7858c4dc21e9cc5968d43662be"} Oct 31 19:39:55.833260 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.833194 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt" event={"ID":"0ba0cb41-a7be-462c-9e92-7dccedd1d92f","Type":"ContainerDied","Data":"6fe550351e235806ea6f50d2a95f4ce4a863f0528ddec2da9bdd752f6c621511"} Oct 31 19:39:55.833260 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.833216 4910 scope.go:122] "RemoveContainer" containerID="e707ec070da09d75ca4ecc29b61cb4444cd40b7858c4dc21e9cc5968d43662be" Oct 31 19:39:55.845800 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.845776 4910 scope.go:122] "RemoveContainer" containerID="e707ec070da09d75ca4ecc29b61cb4444cd40b7858c4dc21e9cc5968d43662be" Oct 31 19:39:55.846114 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:39:55.846088 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e707ec070da09d75ca4ecc29b61cb4444cd40b7858c4dc21e9cc5968d43662be\": container with ID starting with e707ec070da09d75ca4ecc29b61cb4444cd40b7858c4dc21e9cc5968d43662be not found: ID does not exist" containerID="e707ec070da09d75ca4ecc29b61cb4444cd40b7858c4dc21e9cc5968d43662be" Oct 31 19:39:55.846203 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.846124 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e707ec070da09d75ca4ecc29b61cb4444cd40b7858c4dc21e9cc5968d43662be"} err="failed to get container status \"e707ec070da09d75ca4ecc29b61cb4444cd40b7858c4dc21e9cc5968d43662be\": rpc error: code = NotFound desc = could not find container \"e707ec070da09d75ca4ecc29b61cb4444cd40b7858c4dc21e9cc5968d43662be\": container with ID starting with e707ec070da09d75ca4ecc29b61cb4444cd40b7858c4dc21e9cc5968d43662be not found: ID does not exist" Oct 31 19:39:55.861137 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.861115 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt"] Oct 31 19:39:55.866867 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:55.866846 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-jg8gt"] Oct 31 19:39:56.034961 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:56.034832 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3899824-6ld2k"] Oct 31 19:39:56.045427 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:56.045353 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3899824-6ld2k"] Oct 31 19:39:56.052655 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:56.052619 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c562c27-g57ct"] Oct 31 19:39:56.059535 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:56.059510 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c562c27-g57ct"] Oct 31 19:39:57.757313 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:57.757280 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="028abb1d-a374-4439-99d6-b0e13ed729dc" path="/var/lib/kubelet/pods/028abb1d-a374-4439-99d6-b0e13ed729dc/volumes" Oct 31 19:39:57.757800 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:57.757786 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ba0cb41-a7be-462c-9e92-7dccedd1d92f" path="/var/lib/kubelet/pods/0ba0cb41-a7be-462c-9e92-7dccedd1d92f/volumes" Oct 31 19:39:57.758223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:39:57.758210 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f259189-9a2f-4b33-9fd6-b9595ec217ef" path="/var/lib/kubelet/pods/2f259189-9a2f-4b33-9fd6-b9595ec217ef/volumes" Oct 31 19:40:02.673067 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:02.673029 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-t92sd"] Oct 31 19:40:02.674224 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:02.674198 4910 cpu_manager.go:414] "containerMap: removing container" podUID="0ba0cb41-a7be-462c-9e92-7dccedd1d92f" containerName="istio-proxy" Oct 31 19:40:02.674412 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:02.674386 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ba0cb41-a7be-462c-9e92-7dccedd1d92f" containerName="istio-proxy" Oct 31 19:40:02.674620 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:02.674461 4910 cpu_manager.go:414] "containerMap: removing container" podUID="cc7e604e-aa70-438f-9fbc-ee824a47732d" containerName="revoke-keys" Oct 31 19:40:02.674620 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:02.674470 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc7e604e-aa70-438f-9fbc-ee824a47732d" containerName="revoke-keys" Oct 31 19:40:02.674809 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:02.674650 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ba0cb41-a7be-462c-9e92-7dccedd1d92f" containerName="istio-proxy" Oct 31 19:40:02.674809 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:02.674673 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc7e604e-aa70-438f-9fbc-ee824a47732d" containerName="revoke-keys" Oct 31 19:40:02.680762 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:02.680734 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-t92sd" Oct 31 19:40:02.687006 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:02.686949 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:40:02.688994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:02.688972 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:40:02.689571 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:02.689550 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-t92sd"] Oct 31 19:40:02.790660 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:02.790625 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqx9h\" (UniqueName: \"kubernetes.io/projected/ad714bbe-05a2-482c-9042-e27caebb62a7-kube-api-access-jqx9h\") pod \"maas-api-revoke-keys-e2e-mt-67adab47-t92sd\" (UID: \"ad714bbe-05a2-482c-9042-e27caebb62a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-t92sd" Oct 31 19:40:02.791021 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:02.790991 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad714bbe-05a2-482c-9042-e27caebb62a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-67adab47-t92sd\" (UID: \"ad714bbe-05a2-482c-9042-e27caebb62a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-t92sd" Oct 31 19:40:02.791166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:02.791148 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad714bbe-05a2-482c-9042-e27caebb62a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-67adab47-t92sd\" (UID: \"ad714bbe-05a2-482c-9042-e27caebb62a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-t92sd" Oct 31 19:40:02.892929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:02.892877 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqx9h\" (UniqueName: \"kubernetes.io/projected/ad714bbe-05a2-482c-9042-e27caebb62a7-kube-api-access-jqx9h\") pod \"maas-api-revoke-keys-e2e-mt-67adab47-t92sd\" (UID: \"ad714bbe-05a2-482c-9042-e27caebb62a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-t92sd" Oct 31 19:40:02.893174 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:02.893153 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad714bbe-05a2-482c-9042-e27caebb62a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-67adab47-t92sd\" (UID: \"ad714bbe-05a2-482c-9042-e27caebb62a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-t92sd" Oct 31 19:40:02.893293 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:02.893279 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad714bbe-05a2-482c-9042-e27caebb62a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-67adab47-t92sd\" (UID: \"ad714bbe-05a2-482c-9042-e27caebb62a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-t92sd" Oct 31 19:40:02.893783 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:02.893762 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad714bbe-05a2-482c-9042-e27caebb62a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-67adab47-t92sd\" (UID: \"ad714bbe-05a2-482c-9042-e27caebb62a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-t92sd" Oct 31 19:40:02.895872 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:02.895848 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad714bbe-05a2-482c-9042-e27caebb62a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-67adab47-t92sd\" (UID: \"ad714bbe-05a2-482c-9042-e27caebb62a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-t92sd" Oct 31 19:40:02.924441 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:02.924375 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqx9h\" (UniqueName: \"kubernetes.io/projected/ad714bbe-05a2-482c-9042-e27caebb62a7-kube-api-access-jqx9h\") pod \"maas-api-revoke-keys-e2e-mt-67adab47-t92sd\" (UID: \"ad714bbe-05a2-482c-9042-e27caebb62a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-t92sd" Oct 31 19:40:03.004883 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:03.004841 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-t92sd" Oct 31 19:40:03.316225 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:40:03.316186 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad714bbe_05a2_482c_9042_e27caebb62a7.slice/crio-de92e2ffe3ba25ed96fef3727f7c8abf9a975fc1027697a70bab13e24fdc3570 WatchSource:0}: Error finding container de92e2ffe3ba25ed96fef3727f7c8abf9a975fc1027697a70bab13e24fdc3570: Status 404 returned error can't find the container with id de92e2ffe3ba25ed96fef3727f7c8abf9a975fc1027697a70bab13e24fdc3570 Oct 31 19:40:03.318092 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:03.318067 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-t92sd"] Oct 31 19:40:03.959517 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:03.959473 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-t92sd" event={"ID":"ad714bbe-05a2-482c-9042-e27caebb62a7","Type":"ContainerStarted","Data":"b8723fa6b341308336772e9d198c21065e4ed08d29c591901b5f09a1434da676"} Oct 31 19:40:03.959930 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:03.959544 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-t92sd" event={"ID":"ad714bbe-05a2-482c-9042-e27caebb62a7","Type":"ContainerStarted","Data":"de92e2ffe3ba25ed96fef3727f7c8abf9a975fc1027697a70bab13e24fdc3570"} Oct 31 19:40:03.979812 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:03.979768 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-t92sd" podStartSLOduration=1.9797549989999998 podStartE2EDuration="1.979754999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:03.979242292 +0000 UTC m=+1238.342062665" watchObservedRunningTime="2026-10-31 19:40:03.979754999 +0000 UTC m=+1238.342575370" Oct 31 19:40:05.020930 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:05.019143 4910 generic.go:356] "Generic (PLEG): container finished" podID="ad714bbe-05a2-482c-9042-e27caebb62a7" containerID="b8723fa6b341308336772e9d198c21065e4ed08d29c591901b5f09a1434da676" exitCode=0 Oct 31 19:40:05.020930 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:05.019377 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-t92sd" event={"ID":"ad714bbe-05a2-482c-9042-e27caebb62a7","Type":"ContainerDied","Data":"b8723fa6b341308336772e9d198c21065e4ed08d29c591901b5f09a1434da676"} Oct 31 19:40:06.235378 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:06.235348 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-t92sd" Oct 31 19:40:06.412569 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:06.412533 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad714bbe-05a2-482c-9042-e27caebb62a7-kube-api-access-jqx9h\" (UniqueName: \"kubernetes.io/projected/ad714bbe-05a2-482c-9042-e27caebb62a7-kube-api-access-jqx9h\") pod \"ad714bbe-05a2-482c-9042-e27caebb62a7\" (UID: \"ad714bbe-05a2-482c-9042-e27caebb62a7\") " Oct 31 19:40:06.413105 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:06.413076 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad714bbe-05a2-482c-9042-e27caebb62a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad714bbe-05a2-482c-9042-e27caebb62a7-maas-api-xks-ca\") pod \"ad714bbe-05a2-482c-9042-e27caebb62a7\" (UID: \"ad714bbe-05a2-482c-9042-e27caebb62a7\") " Oct 31 19:40:06.413318 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:06.413291 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad714bbe-05a2-482c-9042-e27caebb62a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad714bbe-05a2-482c-9042-e27caebb62a7-maas-api-openshift-ca\") pod \"ad714bbe-05a2-482c-9042-e27caebb62a7\" (UID: \"ad714bbe-05a2-482c-9042-e27caebb62a7\") " Oct 31 19:40:06.414933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:06.414891 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad714bbe-05a2-482c-9042-e27caebb62a7-maas-api-openshift-ca" pod "ad714bbe-05a2-482c-9042-e27caebb62a7" (UID: "ad714bbe-05a2-482c-9042-e27caebb62a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:40:06.417328 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:06.417293 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad714bbe-05a2-482c-9042-e27caebb62a7-maas-api-xks-ca" pod "ad714bbe-05a2-482c-9042-e27caebb62a7" (UID: "ad714bbe-05a2-482c-9042-e27caebb62a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:40:06.418646 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:06.418622 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad714bbe-05a2-482c-9042-e27caebb62a7-kube-api-access-jqx9h" pod "ad714bbe-05a2-482c-9042-e27caebb62a7" (UID: "ad714bbe-05a2-482c-9042-e27caebb62a7"). InnerVolumeSpecName "kube-api-access-jqx9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:40:06.516072 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:06.516034 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad714bbe-05a2-482c-9042-e27caebb62a7-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:06.516305 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:06.516278 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad714bbe-05a2-482c-9042-e27caebb62a7-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:06.516305 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:06.516304 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqx9h\" (UniqueName: \"kubernetes.io/projected/ad714bbe-05a2-482c-9042-e27caebb62a7-kube-api-access-jqx9h\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:07.045895 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.045860 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-t92sd" Oct 31 19:40:07.045895 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.045888 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-t92sd" event={"ID":"ad714bbe-05a2-482c-9042-e27caebb62a7","Type":"ContainerDied","Data":"de92e2ffe3ba25ed96fef3727f7c8abf9a975fc1027697a70bab13e24fdc3570"} Oct 31 19:40:07.046149 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.045954 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de92e2ffe3ba25ed96fef3727f7c8abf9a975fc1027697a70bab13e24fdc3570" Oct 31 19:40:07.180611 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.180579 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65"] Oct 31 19:40:07.181363 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.181332 4910 cpu_manager.go:414] "containerMap: removing container" podUID="ad714bbe-05a2-482c-9042-e27caebb62a7" containerName="revoke-keys" Oct 31 19:40:07.181363 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.181353 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad714bbe-05a2-482c-9042-e27caebb62a7" containerName="revoke-keys" Oct 31 19:40:07.181581 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.181484 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad714bbe-05a2-482c-9042-e27caebb62a7" containerName="revoke-keys" Oct 31 19:40:07.185404 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.185381 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.194217 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.193984 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-03dfee-openshift-default-dockercfg-nfkrf\"" Oct 31 19:40:07.224977 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.224944 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65"] Oct 31 19:40:07.370467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.363642 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ca75657-04c9-4283-9059-220e69657f7c-istiod-ca-cert\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.370467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.363696 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-istio-envoy\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.370467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.363735 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg9lv\" (UniqueName: \"kubernetes.io/projected/4ca75657-04c9-4283-9059-220e69657f7c-kube-api-access-kg9lv\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.370467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.363845 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ca75657-04c9-4283-9059-220e69657f7c-istio-token\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.370467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.363902 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-istio-data\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.370467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.363972 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ca75657-04c9-4283-9059-220e69657f7c-istio-podinfo\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.370467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.364063 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-workload-socket\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.370467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.364176 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-workload-certs\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.370467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.364341 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-credential-socket\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.485353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.470341 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-istio-data\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.485353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.470423 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ca75657-04c9-4283-9059-220e69657f7c-istio-podinfo\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.485353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.470510 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-workload-socket\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.485353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.470608 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-workload-certs\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.485353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.470660 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-credential-socket\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.485353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.470804 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ca75657-04c9-4283-9059-220e69657f7c-istiod-ca-cert\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.485353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.470835 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-istio-envoy\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.485353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.470867 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg9lv\" (UniqueName: \"kubernetes.io/projected/4ca75657-04c9-4283-9059-220e69657f7c-kube-api-access-kg9lv\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.485353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.470989 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ca75657-04c9-4283-9059-220e69657f7c-istio-token\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.485353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.471784 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-istio-data\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.485353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.472498 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-credential-socket\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.485353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.472742 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-workload-socket\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.485353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.472996 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-workload-certs\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.485353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.475809 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ca75657-04c9-4283-9059-220e69657f7c-istiod-ca-cert\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.485353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.481471 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-istio-envoy\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.500706 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.500669 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg9lv\" (UniqueName: \"kubernetes.io/projected/4ca75657-04c9-4283-9059-220e69657f7c-kube-api-access-kg9lv\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.512837 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.512802 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ca75657-04c9-4283-9059-220e69657f7c-istio-token\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.526820 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.526785 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ca75657-04c9-4283-9059-220e69657f7c-istio-podinfo\") pod \"e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:07.808986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:07.806390 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:08.036569 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:08.036535 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e7e2625-mgctc"] Oct 31 19:40:08.043538 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:08.043499 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e7e2625-mgctc"] Oct 31 19:40:08.145716 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:08.145590 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65"] Oct 31 19:40:08.149076 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:08.149036 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:40:08.149262 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:08.149244 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:40:08.149406 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:08.149387 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:40:09.069930 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:09.068991 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" event={"ID":"4ca75657-04c9-4283-9059-220e69657f7c","Type":"ContainerStarted","Data":"19c829415c183e4e9f2ad1ef1a25777173810df061bbd00f5d627fb439bf6d37"} Oct 31 19:40:09.069930 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:09.069064 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" event={"ID":"4ca75657-04c9-4283-9059-220e69657f7c","Type":"ContainerStarted","Data":"349c4bf5c4df522fdac0bd963080b381b5fbe75b5d6a798f2dc85edeb185e314"} Oct 31 19:40:09.096413 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:09.096318 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" podStartSLOduration=2.096297023 podStartE2EDuration="2.096297023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:09.093811686 +0000 UTC m=+1243.456632061" watchObservedRunningTime="2026-10-31 19:40:09.096297023 +0000 UTC m=+1243.459117399" Oct 31 19:40:09.757851 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:09.757810 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df36c2ed-5532-4247-872c-54d13d7506a3" path="/var/lib/kubelet/pods/df36c2ed-5532-4247-872c-54d13d7506a3/volumes" Oct 31 19:40:09.807310 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:09.807275 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:09.813240 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:09.813215 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:10.150060 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:10.150023 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:10.152376 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:10.152309 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:40:12.726513 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:40:12.726470 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:40:25.034941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:25.029055 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z"] Oct 31 19:40:25.056941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:25.054530 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9e323f-r2h8z"] Oct 31 19:40:25.693574 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:25.693536 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db61066b-tvtjj"] Oct 31 19:40:25.700103 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:25.700078 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db61066b-tvtjj" Oct 31 19:40:25.713109 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:25.711522 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db61066b-tvtjj"] Oct 31 19:40:25.754407 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:25.754376 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:40:25.762880 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:25.761677 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:40:25.775923 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:25.775860 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93d0a12-24ee-4523-8543-0e247dbaf974-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-db61066b-tvtjj\" (UID: \"b93d0a12-24ee-4523-8543-0e247dbaf974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db61066b-tvtjj" Oct 31 19:40:25.776102 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:25.776049 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93d0a12-24ee-4523-8543-0e247dbaf974-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-db61066b-tvtjj\" (UID: \"b93d0a12-24ee-4523-8543-0e247dbaf974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db61066b-tvtjj" Oct 31 19:40:25.776289 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:25.776177 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls89s\" (UniqueName: \"kubernetes.io/projected/b93d0a12-24ee-4523-8543-0e247dbaf974-kube-api-access-ls89s\") pod \"maas-api-revoke-keys-e2e-mt-db61066b-tvtjj\" (UID: \"b93d0a12-24ee-4523-8543-0e247dbaf974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db61066b-tvtjj" Oct 31 19:40:25.782100 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:25.782073 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc7e604e-aa70-438f-9fbc-ee824a47732d" path="/var/lib/kubelet/pods/cc7e604e-aa70-438f-9fbc-ee824a47732d/volumes" Oct 31 19:40:25.877396 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:25.877358 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93d0a12-24ee-4523-8543-0e247dbaf974-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-db61066b-tvtjj\" (UID: \"b93d0a12-24ee-4523-8543-0e247dbaf974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db61066b-tvtjj" Oct 31 19:40:25.877563 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:25.877489 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93d0a12-24ee-4523-8543-0e247dbaf974-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-db61066b-tvtjj\" (UID: \"b93d0a12-24ee-4523-8543-0e247dbaf974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db61066b-tvtjj" Oct 31 19:40:25.877606 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:25.877572 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls89s\" (UniqueName: \"kubernetes.io/projected/b93d0a12-24ee-4523-8543-0e247dbaf974-kube-api-access-ls89s\") pod \"maas-api-revoke-keys-e2e-mt-db61066b-tvtjj\" (UID: \"b93d0a12-24ee-4523-8543-0e247dbaf974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db61066b-tvtjj" Oct 31 19:40:25.878159 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:25.878105 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93d0a12-24ee-4523-8543-0e247dbaf974-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-db61066b-tvtjj\" (UID: \"b93d0a12-24ee-4523-8543-0e247dbaf974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db61066b-tvtjj" Oct 31 19:40:25.880495 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:25.880469 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93d0a12-24ee-4523-8543-0e247dbaf974-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-db61066b-tvtjj\" (UID: \"b93d0a12-24ee-4523-8543-0e247dbaf974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db61066b-tvtjj" Oct 31 19:40:25.890573 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:25.890545 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls89s\" (UniqueName: \"kubernetes.io/projected/b93d0a12-24ee-4523-8543-0e247dbaf974-kube-api-access-ls89s\") pod \"maas-api-revoke-keys-e2e-mt-db61066b-tvtjj\" (UID: \"b93d0a12-24ee-4523-8543-0e247dbaf974\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db61066b-tvtjj" Oct 31 19:40:26.076886 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:26.076856 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db61066b-tvtjj" Oct 31 19:40:26.416979 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:26.415936 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db61066b-tvtjj"] Oct 31 19:40:27.398973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:27.398936 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db61066b-tvtjj" event={"ID":"b93d0a12-24ee-4523-8543-0e247dbaf974","Type":"ContainerStarted","Data":"a9b25acd432e61b4abcd5337f6e9d9957192dc3687df8c22a65517af0e656ea8"} Oct 31 19:40:27.399656 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:27.399004 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db61066b-tvtjj" event={"ID":"b93d0a12-24ee-4523-8543-0e247dbaf974","Type":"ContainerStarted","Data":"551b3da916a7a61241644c0f17c2e8a90b717cac099eb8ba2319564c1aaaafcb"} Oct 31 19:40:27.421903 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:27.421821 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db61066b-tvtjj" podStartSLOduration=2.421804192 podStartE2EDuration="2.421804192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:27.41878016 +0000 UTC m=+1261.781600533" watchObservedRunningTime="2026-10-31 19:40:27.421804192 +0000 UTC m=+1261.784624567" Oct 31 19:40:27.623842 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:27.623799 4910 scope.go:122] "RemoveContainer" containerID="33e3a4b3860a0f884ff227659651a566fe1542190911e67cdf70bbfab0273e7e" Oct 31 19:40:27.643784 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:27.643758 4910 scope.go:122] "RemoveContainer" containerID="c2f4d0d3073e969052740a03602ab51fac3a9610672b19f5d0f8363267893540" Oct 31 19:40:27.656210 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:27.656182 4910 scope.go:122] "RemoveContainer" containerID="ba0aad91581763dfba6e46119cdfc33ba2b5ddad4751e7e86176400a1ee0ccdd" Oct 31 19:40:27.668427 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:27.668403 4910 scope.go:122] "RemoveContainer" containerID="3cb74bdac143ca1e096dd4e0df2a4d2d30fa784688c3d76864cc2e499f40d728" Oct 31 19:40:27.678410 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:27.678387 4910 scope.go:122] "RemoveContainer" containerID="d1bb99c3b798c0c0ab9d27b687407b50c7734a6f507f1c4266e8330a1766c6e3" Oct 31 19:40:27.688463 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:27.688442 4910 scope.go:122] "RemoveContainer" containerID="d5d3b47675d29dd99f31e705fdc12f4dd76a93c8931d7f5b63eab2edbbaa35ef" Oct 31 19:40:27.698008 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:27.697988 4910 scope.go:122] "RemoveContainer" containerID="3abdce930a26dbb5aadcee505155eec753702274e944d78d01bedf882d6ebeeb" Oct 31 19:40:27.707669 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:27.707648 4910 scope.go:122] "RemoveContainer" containerID="16b53801741ca46062ba8201305dcc0c6415468863f5e109a9e89d6cdc9857d9" Oct 31 19:40:27.717470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:27.717451 4910 scope.go:122] "RemoveContainer" containerID="95ea8ea30bfba6477dbd6b2b951be76b32ab6cb8ada82cbab01e5fbe5d8b10ee" Oct 31 19:40:27.731004 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:27.730977 4910 scope.go:122] "RemoveContainer" containerID="e3ead7565de785b5853be4d8d7152ffec01086847aa022a6ec11f236468a0db9" Oct 31 19:40:28.474898 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:28.469191 4910 generic.go:356] "Generic (PLEG): container finished" podID="b93d0a12-24ee-4523-8543-0e247dbaf974" containerID="a9b25acd432e61b4abcd5337f6e9d9957192dc3687df8c22a65517af0e656ea8" exitCode=0 Oct 31 19:40:28.474898 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:28.469315 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db61066b-tvtjj" event={"ID":"b93d0a12-24ee-4523-8543-0e247dbaf974","Type":"ContainerDied","Data":"a9b25acd432e61b4abcd5337f6e9d9957192dc3687df8c22a65517af0e656ea8"} Oct 31 19:40:28.709172 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:28.708583 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-t92sd"] Oct 31 19:40:28.716024 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:28.715993 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67adab47-t92sd"] Oct 31 19:40:29.663675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:29.663646 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db61066b-tvtjj" Oct 31 19:40:29.841399 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:29.829299 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b93d0a12-24ee-4523-8543-0e247dbaf974-kube-api-access-ls89s\" (UniqueName: \"kubernetes.io/projected/b93d0a12-24ee-4523-8543-0e247dbaf974-kube-api-access-ls89s\") pod \"b93d0a12-24ee-4523-8543-0e247dbaf974\" (UID: \"b93d0a12-24ee-4523-8543-0e247dbaf974\") " Oct 31 19:40:29.841399 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:29.829439 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b93d0a12-24ee-4523-8543-0e247dbaf974-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93d0a12-24ee-4523-8543-0e247dbaf974-maas-api-openshift-ca\") pod \"b93d0a12-24ee-4523-8543-0e247dbaf974\" (UID: \"b93d0a12-24ee-4523-8543-0e247dbaf974\") " Oct 31 19:40:29.841399 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:29.829604 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b93d0a12-24ee-4523-8543-0e247dbaf974-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93d0a12-24ee-4523-8543-0e247dbaf974-maas-api-xks-ca\") pod \"b93d0a12-24ee-4523-8543-0e247dbaf974\" (UID: \"b93d0a12-24ee-4523-8543-0e247dbaf974\") " Oct 31 19:40:29.971297 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:29.857235 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b93d0a12-24ee-4523-8543-0e247dbaf974-maas-api-openshift-ca" pod "b93d0a12-24ee-4523-8543-0e247dbaf974" (UID: "b93d0a12-24ee-4523-8543-0e247dbaf974"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:40:29.971297 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:29.968312 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b93d0a12-24ee-4523-8543-0e247dbaf974-maas-api-xks-ca" pod "b93d0a12-24ee-4523-8543-0e247dbaf974" (UID: "b93d0a12-24ee-4523-8543-0e247dbaf974"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:40:29.971943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:29.971877 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93d0a12-24ee-4523-8543-0e247dbaf974-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:29.972092 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:29.971949 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93d0a12-24ee-4523-8543-0e247dbaf974-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:30.041972 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.028389 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad714bbe-05a2-482c-9042-e27caebb62a7" path="/var/lib/kubelet/pods/ad714bbe-05a2-482c-9042-e27caebb62a7/volumes" Oct 31 19:40:30.093652 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.082850 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93d0a12-24ee-4523-8543-0e247dbaf974-kube-api-access-ls89s" pod "b93d0a12-24ee-4523-8543-0e247dbaf974" (UID: "b93d0a12-24ee-4523-8543-0e247dbaf974"). InnerVolumeSpecName "kube-api-access-ls89s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:40:30.101042 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.099218 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg"] Oct 31 19:40:30.101042 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.100272 4910 cpu_manager.go:414] "containerMap: removing container" podUID="b93d0a12-24ee-4523-8543-0e247dbaf974" containerName="revoke-keys" Oct 31 19:40:30.101042 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.100293 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b93d0a12-24ee-4523-8543-0e247dbaf974" containerName="revoke-keys" Oct 31 19:40:30.101042 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.100513 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="b93d0a12-24ee-4523-8543-0e247dbaf974" containerName="revoke-keys" Oct 31 19:40:30.136475 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.136439 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg"] Oct 31 19:40:30.136667 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.136584 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.142070 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.142046 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d4b21c-openshift-default-dockercfg-k8zbt\"" Oct 31 19:40:30.183782 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.183746 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.183948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.183791 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.183948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.183883 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7def544-b959-41db-a2b3-a66b68a81d10-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.184075 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.183948 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.184120 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.184097 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7def544-b959-41db-a2b3-a66b68a81d10-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.184247 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.184224 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.184292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.184257 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7def544-b959-41db-a2b3-a66b68a81d10-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.184344 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.184302 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.184439 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.184417 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn9fh\" (UniqueName: \"kubernetes.io/projected/d7def544-b959-41db-a2b3-a66b68a81d10-kube-api-access-bn9fh\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.184536 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.184524 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls89s\" (UniqueName: \"kubernetes.io/projected/b93d0a12-24ee-4523-8543-0e247dbaf974-kube-api-access-ls89s\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:30.285637 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.285594 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.285841 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.285654 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.285841 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.285729 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7def544-b959-41db-a2b3-a66b68a81d10-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.285841 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.285808 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.286107 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.285871 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7def544-b959-41db-a2b3-a66b68a81d10-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.286107 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.286003 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.286107 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.286036 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7def544-b959-41db-a2b3-a66b68a81d10-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.286107 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.286085 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.286320 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.286126 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.286320 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.286192 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn9fh\" (UniqueName: \"kubernetes.io/projected/d7def544-b959-41db-a2b3-a66b68a81d10-kube-api-access-bn9fh\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.286320 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.286216 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.286479 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.286368 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.286479 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.286428 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.286588 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.286503 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7def544-b959-41db-a2b3-a66b68a81d10-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.288055 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.288033 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.288926 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.288879 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7def544-b959-41db-a2b3-a66b68a81d10-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.302160 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.302136 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7def544-b959-41db-a2b3-a66b68a81d10-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.319139 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.319113 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn9fh\" (UniqueName: \"kubernetes.io/projected/d7def544-b959-41db-a2b3-a66b68a81d10-kube-api-access-bn9fh\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.453981 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.453883 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:30.496929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.496882 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db61066b-tvtjj" event={"ID":"b93d0a12-24ee-4523-8543-0e247dbaf974","Type":"ContainerDied","Data":"551b3da916a7a61241644c0f17c2e8a90b717cac099eb8ba2319564c1aaaafcb"} Oct 31 19:40:30.497165 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.497054 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db61066b-tvtjj" Oct 31 19:40:30.498025 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.497998 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="551b3da916a7a61241644c0f17c2e8a90b717cac099eb8ba2319564c1aaaafcb" Oct 31 19:40:30.772276 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:40:30.772239 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7def544_b959_41db_a2b3_a66b68a81d10.slice/crio-52029cd5dd830f32208a6165d80c956804b6dfd9f6160f1a9cce8939bd7ed095 WatchSource:0}: Error finding container 52029cd5dd830f32208a6165d80c956804b6dfd9f6160f1a9cce8939bd7ed095: Status 404 returned error can't find the container with id 52029cd5dd830f32208a6165d80c956804b6dfd9f6160f1a9cce8939bd7ed095 Oct 31 19:40:30.773618 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.773591 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg"] Oct 31 19:40:30.775648 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.775616 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:40:30.775738 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.775678 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:40:30.775738 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:30.775704 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:40:31.516709 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:31.516666 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" event={"ID":"d7def544-b959-41db-a2b3-a66b68a81d10","Type":"ContainerStarted","Data":"5f4dccb6d6a00184dbb07aa13cefb8695cbe3993d1ef9ab2bde648081399a098"} Oct 31 19:40:31.516924 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:31.516735 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" event={"ID":"d7def544-b959-41db-a2b3-a66b68a81d10","Type":"ContainerStarted","Data":"52029cd5dd830f32208a6165d80c956804b6dfd9f6160f1a9cce8939bd7ed095"} Oct 31 19:40:31.552491 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:31.550957 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" podStartSLOduration=2.550941786 podStartE2EDuration="2.550941786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:31.547607101 +0000 UTC m=+1265.910427474" watchObservedRunningTime="2026-10-31 19:40:31.550941786 +0000 UTC m=+1265.913762158" Oct 31 19:40:32.457929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:32.454547 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:32.457929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:32.457585 4910 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 19:40:32.457929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:32.457666 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" podUID="d7def544-b959-41db-a2b3-a66b68a81d10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 19:40:33.467388 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:33.455297 4910 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 19:40:33.467388 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:33.455392 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" podUID="d7def544-b959-41db-a2b3-a66b68a81d10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 19:40:34.488824 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:34.487082 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg"] Oct 31 19:40:34.488824 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:34.487332 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" podUID="71387fb8-12d0-4620-a56c-7b4cd6ba554e" containerName="istio-proxy" containerID="cri-o://f167e66498f2b6403f6f04252b222a5721b5ad55853f3338d44400fba5644a10" gracePeriod=30 Oct 31 19:40:34.494467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:34.494430 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s"] Oct 31 19:40:34.494713 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:34.494681 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" podUID="638cecc0-b837-4dfb-9651-793b6f2986b2" containerName="istio-proxy" containerID="cri-o://098e5ca4398fca1404631c9535d37d81fc8b88296304452244679b31867644aa" gracePeriod=30 Oct 31 19:40:34.496087 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:34.496041 4910 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 19:40:34.496206 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:34.496087 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" podUID="d7def544-b959-41db-a2b3-a66b68a81d10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 19:40:34.510497 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:34.501749 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj"] Oct 31 19:40:34.510497 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:34.501969 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" podUID="1540ff89-ac68-464d-b39f-c01ef4ebd62c" containerName="istio-proxy" containerID="cri-o://38382a8c33bdc1e0fdbdd75510fc7669f62aadc2001034561d465318f3aba8ec" gracePeriod=30 Oct 31 19:40:34.671871 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:34.669459 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4"] Oct 31 19:40:34.679721 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:34.679689 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4"] Oct 31 19:40:34.679942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:34.679804 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4" Oct 31 19:40:34.684009 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:34.683986 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:40:34.684170 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:34.684155 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:40:34.800613 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:34.800521 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ac06c4-37b4-4da9-8c00-0c91d0d4143b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4\" (UID: \"20ac06c4-37b4-4da9-8c00-0c91d0d4143b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4" Oct 31 19:40:34.800833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:34.800805 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvxmt\" (UniqueName: \"kubernetes.io/projected/20ac06c4-37b4-4da9-8c00-0c91d0d4143b-kube-api-access-vvxmt\") pod \"maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4\" (UID: \"20ac06c4-37b4-4da9-8c00-0c91d0d4143b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4" Oct 31 19:40:34.801032 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:34.801014 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ac06c4-37b4-4da9-8c00-0c91d0d4143b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4\" (UID: \"20ac06c4-37b4-4da9-8c00-0c91d0d4143b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4" Oct 31 19:40:34.903038 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:34.902998 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ac06c4-37b4-4da9-8c00-0c91d0d4143b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4\" (UID: \"20ac06c4-37b4-4da9-8c00-0c91d0d4143b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4" Oct 31 19:40:34.903238 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:34.903216 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvxmt\" (UniqueName: \"kubernetes.io/projected/20ac06c4-37b4-4da9-8c00-0c91d0d4143b-kube-api-access-vvxmt\") pod \"maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4\" (UID: \"20ac06c4-37b4-4da9-8c00-0c91d0d4143b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4" Oct 31 19:40:34.903324 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:34.903310 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ac06c4-37b4-4da9-8c00-0c91d0d4143b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4\" (UID: \"20ac06c4-37b4-4da9-8c00-0c91d0d4143b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4" Oct 31 19:40:34.903861 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:34.903826 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ac06c4-37b4-4da9-8c00-0c91d0d4143b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4\" (UID: \"20ac06c4-37b4-4da9-8c00-0c91d0d4143b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4" Oct 31 19:40:34.906195 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:34.906173 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ac06c4-37b4-4da9-8c00-0c91d0d4143b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4\" (UID: \"20ac06c4-37b4-4da9-8c00-0c91d0d4143b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4" Oct 31 19:40:34.916015 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:34.915993 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvxmt\" (UniqueName: \"kubernetes.io/projected/20ac06c4-37b4-4da9-8c00-0c91d0d4143b-kube-api-access-vvxmt\") pod \"maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4\" (UID: \"20ac06c4-37b4-4da9-8c00-0c91d0d4143b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4" Oct 31 19:40:35.008624 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:35.006823 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4" Oct 31 19:40:35.682300 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:35.682256 4910 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 19:40:35.682758 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:35.682331 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" podUID="d7def544-b959-41db-a2b3-a66b68a81d10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 19:40:36.017578 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:36.017546 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4"] Oct 31 19:40:36.455603 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:36.455565 4910 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 19:40:36.455887 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:36.455860 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" podUID="d7def544-b959-41db-a2b3-a66b68a81d10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 19:40:36.851086 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:36.851040 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4" event={"ID":"20ac06c4-37b4-4da9-8c00-0c91d0d4143b","Type":"ContainerStarted","Data":"97ca3b39271a1a9e89a91d6b3ee4fd054359438202062afe357e000d266128f8"} Oct 31 19:40:36.851708 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:36.851663 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4" event={"ID":"20ac06c4-37b4-4da9-8c00-0c91d0d4143b","Type":"ContainerStarted","Data":"4dd43765943f1101b5b00c21906e4cbf03ba3e9ecf60f638574a3090ba7312b2"} Oct 31 19:40:37.455936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:37.455531 4910 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 19:40:37.455936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:37.455605 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" podUID="d7def544-b959-41db-a2b3-a66b68a81d10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 19:40:37.863872 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:37.863832 4910 generic.go:356] "Generic (PLEG): container finished" podID="20ac06c4-37b4-4da9-8c00-0c91d0d4143b" containerID="97ca3b39271a1a9e89a91d6b3ee4fd054359438202062afe357e000d266128f8" exitCode=0 Oct 31 19:40:37.864505 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:37.864215 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4" event={"ID":"20ac06c4-37b4-4da9-8c00-0c91d0d4143b","Type":"ContainerDied","Data":"97ca3b39271a1a9e89a91d6b3ee4fd054359438202062afe357e000d266128f8"} Oct 31 19:40:38.004996 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:38.004956 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:40:38.005207 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:38.005034 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" podUID="71387fb8-12d0-4620-a56c-7b4cd6ba554e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:40:38.493014 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:38.492984 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:38.493520 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:38.493493 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:38.494979 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:38.494955 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:40:38.568694 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:38.563985 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:40:38.568694 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:38.564071 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" podUID="1540ff89-ac68-464d-b39f-c01ef4ebd62c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:40:39.069970 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:39.069938 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4" Oct 31 19:40:39.111512 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:39.111478 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20ac06c4-37b4-4da9-8c00-0c91d0d4143b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ac06c4-37b4-4da9-8c00-0c91d0d4143b-maas-api-xks-ca\") pod \"20ac06c4-37b4-4da9-8c00-0c91d0d4143b\" (UID: \"20ac06c4-37b4-4da9-8c00-0c91d0d4143b\") " Oct 31 19:40:39.111819 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:39.111796 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20ac06c4-37b4-4da9-8c00-0c91d0d4143b-kube-api-access-vvxmt\" (UniqueName: \"kubernetes.io/projected/20ac06c4-37b4-4da9-8c00-0c91d0d4143b-kube-api-access-vvxmt\") pod \"20ac06c4-37b4-4da9-8c00-0c91d0d4143b\" (UID: \"20ac06c4-37b4-4da9-8c00-0c91d0d4143b\") " Oct 31 19:40:39.112211 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:39.112189 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20ac06c4-37b4-4da9-8c00-0c91d0d4143b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ac06c4-37b4-4da9-8c00-0c91d0d4143b-maas-api-openshift-ca\") pod \"20ac06c4-37b4-4da9-8c00-0c91d0d4143b\" (UID: \"20ac06c4-37b4-4da9-8c00-0c91d0d4143b\") " Oct 31 19:40:39.112659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:39.112617 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20ac06c4-37b4-4da9-8c00-0c91d0d4143b-maas-api-openshift-ca" pod "20ac06c4-37b4-4da9-8c00-0c91d0d4143b" (UID: "20ac06c4-37b4-4da9-8c00-0c91d0d4143b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:40:39.113315 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:39.113288 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20ac06c4-37b4-4da9-8c00-0c91d0d4143b-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:39.126028 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:39.114607 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20ac06c4-37b4-4da9-8c00-0c91d0d4143b-maas-api-xks-ca" pod "20ac06c4-37b4-4da9-8c00-0c91d0d4143b" (UID: "20ac06c4-37b4-4da9-8c00-0c91d0d4143b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:40:39.126028 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:39.114725 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ac06c4-37b4-4da9-8c00-0c91d0d4143b-kube-api-access-vvxmt" pod "20ac06c4-37b4-4da9-8c00-0c91d0d4143b" (UID: "20ac06c4-37b4-4da9-8c00-0c91d0d4143b"). InnerVolumeSpecName "kube-api-access-vvxmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:40:39.215223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:39.215185 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20ac06c4-37b4-4da9-8c00-0c91d0d4143b-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:39.215223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:39.215227 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvxmt\" (UniqueName: \"kubernetes.io/projected/20ac06c4-37b4-4da9-8c00-0c91d0d4143b-kube-api-access-vvxmt\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:39.240545 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:39.239661 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98"] Oct 31 19:40:39.240545 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:39.239967 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98" podUID="505b718c-f2ab-4f6d-892a-ed614bc2335d" containerName="maas-api" containerID="cri-o://42eef546a273aabac292f803e28b8ec7af7fb7361081b16f2ca1765fdbd10232" gracePeriod=30 Oct 31 19:40:40.033144 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.033062 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db61066b-tvtjj"] Oct 31 19:40:40.033395 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.033377 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-db61066b-tvtjj"] Oct 31 19:40:40.089064 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.088678 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98" Oct 31 19:40:40.094744 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.089734 4910 generic.go:356] "Generic (PLEG): container finished" podID="71387fb8-12d0-4620-a56c-7b4cd6ba554e" containerID="f167e66498f2b6403f6f04252b222a5721b5ad55853f3338d44400fba5644a10" exitCode=0 Oct 31 19:40:40.094744 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.089775 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" event={"ID":"71387fb8-12d0-4620-a56c-7b4cd6ba554e","Type":"ContainerDied","Data":"f167e66498f2b6403f6f04252b222a5721b5ad55853f3338d44400fba5644a10"} Oct 31 19:40:40.094744 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.092508 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4" event={"ID":"20ac06c4-37b4-4da9-8c00-0c91d0d4143b","Type":"ContainerDied","Data":"4dd43765943f1101b5b00c21906e4cbf03ba3e9ecf60f638574a3090ba7312b2"} Oct 31 19:40:40.094744 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.092556 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dd43765943f1101b5b00c21906e4cbf03ba3e9ecf60f638574a3090ba7312b2" Oct 31 19:40:40.094744 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.092412 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4" Oct 31 19:40:40.101433 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.101402 4910 generic.go:356] "Generic (PLEG): container finished" podID="638cecc0-b837-4dfb-9651-793b6f2986b2" containerID="098e5ca4398fca1404631c9535d37d81fc8b88296304452244679b31867644aa" exitCode=0 Oct 31 19:40:40.101651 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.101570 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" event={"ID":"638cecc0-b837-4dfb-9651-793b6f2986b2","Type":"ContainerDied","Data":"098e5ca4398fca1404631c9535d37d81fc8b88296304452244679b31867644aa"} Oct 31 19:40:40.104834 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.104811 4910 generic.go:356] "Generic (PLEG): container finished" podID="505b718c-f2ab-4f6d-892a-ed614bc2335d" containerID="42eef546a273aabac292f803e28b8ec7af7fb7361081b16f2ca1765fdbd10232" exitCode=0 Oct 31 19:40:40.105206 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.105189 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98" Oct 31 19:40:40.105472 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.105453 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98" event={"ID":"505b718c-f2ab-4f6d-892a-ed614bc2335d","Type":"ContainerDied","Data":"42eef546a273aabac292f803e28b8ec7af7fb7361081b16f2ca1765fdbd10232"} Oct 31 19:40:40.105548 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.105506 4910 scope.go:122] "RemoveContainer" containerID="42eef546a273aabac292f803e28b8ec7af7fb7361081b16f2ca1765fdbd10232" Oct 31 19:40:40.113314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.112351 4910 generic.go:356] "Generic (PLEG): container finished" podID="1540ff89-ac68-464d-b39f-c01ef4ebd62c" containerID="38382a8c33bdc1e0fdbdd75510fc7669f62aadc2001034561d465318f3aba8ec" exitCode=0 Oct 31 19:40:40.113314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.112875 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" event={"ID":"1540ff89-ac68-464d-b39f-c01ef4ebd62c","Type":"ContainerDied","Data":"38382a8c33bdc1e0fdbdd75510fc7669f62aadc2001034561d465318f3aba8ec"} Oct 31 19:40:40.117566 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.117541 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/505b718c-f2ab-4f6d-892a-ed614bc2335d-kube-api-access-lz6ht\" (UniqueName: \"kubernetes.io/projected/505b718c-f2ab-4f6d-892a-ed614bc2335d-kube-api-access-lz6ht\") pod \"505b718c-f2ab-4f6d-892a-ed614bc2335d\" (UID: \"505b718c-f2ab-4f6d-892a-ed614bc2335d\") " Oct 31 19:40:40.118285 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.118263 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/505b718c-f2ab-4f6d-892a-ed614bc2335d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/505b718c-f2ab-4f6d-892a-ed614bc2335d-openshift-service-ca\") pod \"505b718c-f2ab-4f6d-892a-ed614bc2335d\" (UID: \"505b718c-f2ab-4f6d-892a-ed614bc2335d\") " Oct 31 19:40:40.124031 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.122015 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/505b718c-f2ab-4f6d-892a-ed614bc2335d-openshift-service-ca" pod "505b718c-f2ab-4f6d-892a-ed614bc2335d" (UID: "505b718c-f2ab-4f6d-892a-ed614bc2335d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:40:40.124399 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.124367 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/505b718c-f2ab-4f6d-892a-ed614bc2335d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/505b718c-f2ab-4f6d-892a-ed614bc2335d-metrics-certs\") pod \"505b718c-f2ab-4f6d-892a-ed614bc2335d\" (UID: \"505b718c-f2ab-4f6d-892a-ed614bc2335d\") " Oct 31 19:40:40.124614 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.124599 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/505b718c-f2ab-4f6d-892a-ed614bc2335d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/505b718c-f2ab-4f6d-892a-ed614bc2335d-maas-api-tls\") pod \"505b718c-f2ab-4f6d-892a-ed614bc2335d\" (UID: \"505b718c-f2ab-4f6d-892a-ed614bc2335d\") " Oct 31 19:40:40.125630 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.125442 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505b718c-f2ab-4f6d-892a-ed614bc2335d-kube-api-access-lz6ht" pod "505b718c-f2ab-4f6d-892a-ed614bc2335d" (UID: "505b718c-f2ab-4f6d-892a-ed614bc2335d"). InnerVolumeSpecName "kube-api-access-lz6ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:40:40.145800 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.134171 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/505b718c-f2ab-4f6d-892a-ed614bc2335d-metrics-certs" pod "505b718c-f2ab-4f6d-892a-ed614bc2335d" (UID: "505b718c-f2ab-4f6d-892a-ed614bc2335d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:40:40.147455 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.147410 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/505b718c-f2ab-4f6d-892a-ed614bc2335d-maas-api-tls" pod "505b718c-f2ab-4f6d-892a-ed614bc2335d" (UID: "505b718c-f2ab-4f6d-892a-ed614bc2335d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:40:40.171383 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.171346 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/505b718c-f2ab-4f6d-892a-ed614bc2335d-maas-api-tls\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.171630 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.171610 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz6ht\" (UniqueName: \"kubernetes.io/projected/505b718c-f2ab-4f6d-892a-ed614bc2335d-kube-api-access-lz6ht\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.171815 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.171799 4910 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/505b718c-f2ab-4f6d-892a-ed614bc2335d-openshift-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.171952 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.171937 4910 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/505b718c-f2ab-4f6d-892a-ed614bc2335d-metrics-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.283316 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.283234 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:40:40.360802 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.360423 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:40:40.384982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.375536 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-workload-socket\") pod \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " Oct 31 19:40:40.384982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.375807 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-workload-socket" pod "1540ff89-ac68-464d-b39f-c01ef4ebd62c" (UID: "1540ff89-ac68-464d-b39f-c01ef4ebd62c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:40:40.384982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.376274 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-workload-certs" pod "1540ff89-ac68-464d-b39f-c01ef4ebd62c" (UID: "1540ff89-ac68-464d-b39f-c01ef4ebd62c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:40:40.384982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.375905 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-workload-certs\") pod \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " Oct 31 19:40:40.384982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.376358 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-token\" (UniqueName: \"kubernetes.io/projected/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-token\") pod \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " Oct 31 19:40:40.384982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.377385 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1540ff89-ac68-464d-b39f-c01ef4ebd62c-kube-api-access-bvrv2\" (UniqueName: \"kubernetes.io/projected/1540ff89-ac68-464d-b39f-c01ef4ebd62c-kube-api-access-bvrv2\") pod \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " Oct 31 19:40:40.384982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.377754 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istiod-ca-cert\") pod \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " Oct 31 19:40:40.384982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.378174 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istiod-ca-cert" pod "1540ff89-ac68-464d-b39f-c01ef4ebd62c" (UID: "1540ff89-ac68-464d-b39f-c01ef4ebd62c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:40:40.384982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.378300 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-data\") pod \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " Oct 31 19:40:40.384982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.378619 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-data" pod "1540ff89-ac68-464d-b39f-c01ef4ebd62c" (UID: "1540ff89-ac68-464d-b39f-c01ef4ebd62c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:40:40.384982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.378677 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-podinfo\") pod \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " Oct 31 19:40:40.384982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.378728 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-token" pod "1540ff89-ac68-464d-b39f-c01ef4ebd62c" (UID: "1540ff89-ac68-464d-b39f-c01ef4ebd62c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:40:40.384982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.379155 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-envoy\") pod \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " Oct 31 19:40:40.384982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.379245 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-credential-socket\") pod \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\" (UID: \"1540ff89-ac68-464d-b39f-c01ef4ebd62c\") " Oct 31 19:40:40.384982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.379498 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-credential-socket" pod "1540ff89-ac68-464d-b39f-c01ef4ebd62c" (UID: "1540ff89-ac68-464d-b39f-c01ef4ebd62c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:40:40.384982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.380820 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.384982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.380845 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.384982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.380859 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.384982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.380872 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.384982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.380885 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.384982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.380898 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.384982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.381223 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1540ff89-ac68-464d-b39f-c01ef4ebd62c-kube-api-access-bvrv2" pod "1540ff89-ac68-464d-b39f-c01ef4ebd62c" (UID: "1540ff89-ac68-464d-b39f-c01ef4ebd62c"). InnerVolumeSpecName "kube-api-access-bvrv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:40:40.384982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.381433 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-podinfo" pod "1540ff89-ac68-464d-b39f-c01ef4ebd62c" (UID: "1540ff89-ac68-464d-b39f-c01ef4ebd62c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:40:40.384982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.383135 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-envoy" pod "1540ff89-ac68-464d-b39f-c01ef4ebd62c" (UID: "1540ff89-ac68-464d-b39f-c01ef4ebd62c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:40:40.388616 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.388246 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:40:40.448933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.448152 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98"] Oct 31 19:40:40.455267 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.455240 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-db61066b-58d9b4d459-zlw98"] Oct 31 19:40:40.482034 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.482001 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-data\") pod \"638cecc0-b837-4dfb-9651-793b6f2986b2\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " Oct 31 19:40:40.482173 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.482058 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71387fb8-12d0-4620-a56c-7b4cd6ba554e-kube-api-access-8r8hc\" (UniqueName: \"kubernetes.io/projected/71387fb8-12d0-4620-a56c-7b4cd6ba554e-kube-api-access-8r8hc\") pod \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " Oct 31 19:40:40.482173 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.482108 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-workload-socket\") pod \"638cecc0-b837-4dfb-9651-793b6f2986b2\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " Oct 31 19:40:40.482173 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.482134 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/638cecc0-b837-4dfb-9651-793b6f2986b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/638cecc0-b837-4dfb-9651-793b6f2986b2-istiod-ca-cert\") pod \"638cecc0-b837-4dfb-9651-793b6f2986b2\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " Oct 31 19:40:40.482425 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.482171 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-workload-certs\") pod \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " Oct 31 19:40:40.482425 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.482227 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-workload-socket\") pod \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " Oct 31 19:40:40.482425 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.482269 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-data" pod "638cecc0-b837-4dfb-9651-793b6f2986b2" (UID: "638cecc0-b837-4dfb-9651-793b6f2986b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:40:40.482425 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.482340 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-workload-socket" pod "638cecc0-b837-4dfb-9651-793b6f2986b2" (UID: "638cecc0-b837-4dfb-9651-793b6f2986b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:40:40.482425 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.482353 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-podinfo\") pod \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " Oct 31 19:40:40.482704 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.482436 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-data\") pod \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " Oct 31 19:40:40.482704 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.482489 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-envoy\") pod \"638cecc0-b837-4dfb-9651-793b6f2986b2\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " Oct 31 19:40:40.482704 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.482518 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-token\" (UniqueName: \"kubernetes.io/projected/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-token\") pod \"638cecc0-b837-4dfb-9651-793b6f2986b2\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " Oct 31 19:40:40.482704 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.482572 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-token\" (UniqueName: \"kubernetes.io/projected/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-token\") pod \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " Oct 31 19:40:40.482704 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.482595 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istiod-ca-cert\") pod \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " Oct 31 19:40:40.482704 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.482632 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-envoy\") pod \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " Oct 31 19:40:40.482704 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.482681 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-podinfo\") pod \"638cecc0-b837-4dfb-9651-793b6f2986b2\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " Oct 31 19:40:40.483068 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.482726 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-workload-certs\") pod \"638cecc0-b837-4dfb-9651-793b6f2986b2\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " Oct 31 19:40:40.483068 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.482781 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-credential-socket\") pod \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\" (UID: \"71387fb8-12d0-4620-a56c-7b4cd6ba554e\") " Oct 31 19:40:40.483068 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.482801 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/638cecc0-b837-4dfb-9651-793b6f2986b2-kube-api-access-hpgtg\" (UniqueName: \"kubernetes.io/projected/638cecc0-b837-4dfb-9651-793b6f2986b2-kube-api-access-hpgtg\") pod \"638cecc0-b837-4dfb-9651-793b6f2986b2\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " Oct 31 19:40:40.483068 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.482836 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-credential-socket\") pod \"638cecc0-b837-4dfb-9651-793b6f2986b2\" (UID: \"638cecc0-b837-4dfb-9651-793b6f2986b2\") " Oct 31 19:40:40.483068 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.482824 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/638cecc0-b837-4dfb-9651-793b6f2986b2-istiod-ca-cert" pod "638cecc0-b837-4dfb-9651-793b6f2986b2" (UID: "638cecc0-b837-4dfb-9651-793b6f2986b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:40:40.483068 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.482938 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-workload-socket" pod "71387fb8-12d0-4620-a56c-7b4cd6ba554e" (UID: "71387fb8-12d0-4620-a56c-7b4cd6ba554e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:40:40.483467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.483437 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-data" pod "71387fb8-12d0-4620-a56c-7b4cd6ba554e" (UID: "71387fb8-12d0-4620-a56c-7b4cd6ba554e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:40:40.483796 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.483444 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.483896 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.483817 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvrv2\" (UniqueName: \"kubernetes.io/projected/1540ff89-ac68-464d-b39f-c01ef4ebd62c-kube-api-access-bvrv2\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.484041 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.483838 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.484041 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.483967 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/638cecc0-b837-4dfb-9651-793b6f2986b2-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.484041 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.483982 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.484041 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.483997 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.484041 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.484011 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1540ff89-ac68-464d-b39f-c01ef4ebd62c-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.484041 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.483645 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-credential-socket" pod "638cecc0-b837-4dfb-9651-793b6f2986b2" (UID: "638cecc0-b837-4dfb-9651-793b6f2986b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:40:40.484041 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.483878 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-workload-certs" pod "71387fb8-12d0-4620-a56c-7b4cd6ba554e" (UID: "71387fb8-12d0-4620-a56c-7b4cd6ba554e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:40:40.484467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.484441 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istiod-ca-cert" pod "71387fb8-12d0-4620-a56c-7b4cd6ba554e" (UID: "71387fb8-12d0-4620-a56c-7b4cd6ba554e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:40:40.485030 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.485005 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-credential-socket" pod "71387fb8-12d0-4620-a56c-7b4cd6ba554e" (UID: "71387fb8-12d0-4620-a56c-7b4cd6ba554e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:40:40.485233 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.485211 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-workload-certs" pod "638cecc0-b837-4dfb-9651-793b6f2986b2" (UID: "638cecc0-b837-4dfb-9651-793b6f2986b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:40:40.486134 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.485229 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-envoy" pod "638cecc0-b837-4dfb-9651-793b6f2986b2" (UID: "638cecc0-b837-4dfb-9651-793b6f2986b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:40:40.486134 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.485838 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-envoy" pod "71387fb8-12d0-4620-a56c-7b4cd6ba554e" (UID: "71387fb8-12d0-4620-a56c-7b4cd6ba554e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:40:40.486134 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.485836 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-podinfo" pod "638cecc0-b837-4dfb-9651-793b6f2986b2" (UID: "638cecc0-b837-4dfb-9651-793b6f2986b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:40:40.487043 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.486534 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-token" pod "71387fb8-12d0-4620-a56c-7b4cd6ba554e" (UID: "71387fb8-12d0-4620-a56c-7b4cd6ba554e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:40:40.487962 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.487938 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-token" pod "638cecc0-b837-4dfb-9651-793b6f2986b2" (UID: "638cecc0-b837-4dfb-9651-793b6f2986b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:40:40.487962 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.487946 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-podinfo" pod "71387fb8-12d0-4620-a56c-7b4cd6ba554e" (UID: "71387fb8-12d0-4620-a56c-7b4cd6ba554e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:40:40.488099 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.488083 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71387fb8-12d0-4620-a56c-7b4cd6ba554e-kube-api-access-8r8hc" pod "71387fb8-12d0-4620-a56c-7b4cd6ba554e" (UID: "71387fb8-12d0-4620-a56c-7b4cd6ba554e"). InnerVolumeSpecName "kube-api-access-8r8hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:40:40.488827 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.488804 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/638cecc0-b837-4dfb-9651-793b6f2986b2-kube-api-access-hpgtg" pod "638cecc0-b837-4dfb-9651-793b6f2986b2" (UID: "638cecc0-b837-4dfb-9651-793b6f2986b2"). InnerVolumeSpecName "kube-api-access-hpgtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:40:40.590533 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.588660 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.590533 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.588702 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.590533 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.588719 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.590533 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.588732 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.590533 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.588748 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.590533 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.588762 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.590533 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.588775 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.590533 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.588790 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/638cecc0-b837-4dfb-9651-793b6f2986b2-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.590533 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.588803 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.590533 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.588817 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.590533 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.588830 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpgtg\" (UniqueName: \"kubernetes.io/projected/638cecc0-b837-4dfb-9651-793b6f2986b2-kube-api-access-hpgtg\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.590533 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.588844 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/638cecc0-b837-4dfb-9651-793b6f2986b2-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.590533 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.588859 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r8hc\" (UniqueName: \"kubernetes.io/projected/71387fb8-12d0-4620-a56c-7b4cd6ba554e-kube-api-access-8r8hc\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:40.590533 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:40.588873 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71387fb8-12d0-4620-a56c-7b4cd6ba554e-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:41.122752 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:41.122722 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" Oct 31 19:40:41.123386 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:41.122739 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg" event={"ID":"71387fb8-12d0-4620-a56c-7b4cd6ba554e","Type":"ContainerDied","Data":"73f34509f1c6127f6aef35bdae3a71f7ebc24ae228fddabd88fe063c361a9044"} Oct 31 19:40:41.123538 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:41.123434 4910 scope.go:122] "RemoveContainer" containerID="f167e66498f2b6403f6f04252b222a5721b5ad55853f3338d44400fba5644a10" Oct 31 19:40:41.124812 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:41.124741 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" Oct 31 19:40:41.124812 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:41.124784 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s" event={"ID":"638cecc0-b837-4dfb-9651-793b6f2986b2","Type":"ContainerDied","Data":"90d737aaa42a428a0655c62579e9015cdd3743f1dcc3dae288978db6e48b6565"} Oct 31 19:40:41.128431 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:41.128407 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" event={"ID":"1540ff89-ac68-464d-b39f-c01ef4ebd62c","Type":"ContainerDied","Data":"e9aa7a882a703f4275e6e94b22b64aafa941578e41e9e9dd69b21804b76c4eb5"} Oct 31 19:40:41.128550 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:41.128500 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj" Oct 31 19:40:41.138814 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:41.138797 4910 scope.go:122] "RemoveContainer" containerID="098e5ca4398fca1404631c9535d37d81fc8b88296304452244679b31867644aa" Oct 31 19:40:41.149835 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:41.149815 4910 scope.go:122] "RemoveContainer" containerID="38382a8c33bdc1e0fdbdd75510fc7669f62aadc2001034561d465318f3aba8ec" Oct 31 19:40:41.185512 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:41.185481 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj"] Oct 31 19:40:41.191987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:41.191963 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-vc4dj"] Oct 31 19:40:41.228269 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:41.228234 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg"] Oct 31 19:40:41.240804 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:41.240771 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4s2jg"] Oct 31 19:40:41.260562 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:41.260529 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s"] Oct 31 19:40:41.307206 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:41.307176 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-4hh2s"] Oct 31 19:40:41.757855 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:41.757780 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1540ff89-ac68-464d-b39f-c01ef4ebd62c" path="/var/lib/kubelet/pods/1540ff89-ac68-464d-b39f-c01ef4ebd62c/volumes" Oct 31 19:40:41.758387 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:41.758372 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="505b718c-f2ab-4f6d-892a-ed614bc2335d" path="/var/lib/kubelet/pods/505b718c-f2ab-4f6d-892a-ed614bc2335d/volumes" Oct 31 19:40:41.758850 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:41.758831 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="638cecc0-b837-4dfb-9651-793b6f2986b2" path="/var/lib/kubelet/pods/638cecc0-b837-4dfb-9651-793b6f2986b2/volumes" Oct 31 19:40:41.759665 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:41.759650 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71387fb8-12d0-4620-a56c-7b4cd6ba554e" path="/var/lib/kubelet/pods/71387fb8-12d0-4620-a56c-7b4cd6ba554e/volumes" Oct 31 19:40:41.760109 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:41.760094 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b93d0a12-24ee-4523-8543-0e247dbaf974" path="/var/lib/kubelet/pods/b93d0a12-24ee-4523-8543-0e247dbaf974/volumes" Oct 31 19:40:42.740303 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:40:42.740266 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:40:44.163076 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.163038 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq"] Oct 31 19:40:44.163944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.163919 4910 cpu_manager.go:414] "containerMap: removing container" podUID="71387fb8-12d0-4620-a56c-7b4cd6ba554e" containerName="istio-proxy" Oct 31 19:40:44.163944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.163943 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71387fb8-12d0-4620-a56c-7b4cd6ba554e" containerName="istio-proxy" Oct 31 19:40:44.164134 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.163973 4910 cpu_manager.go:414] "containerMap: removing container" podUID="1540ff89-ac68-464d-b39f-c01ef4ebd62c" containerName="istio-proxy" Oct 31 19:40:44.164134 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.163982 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1540ff89-ac68-464d-b39f-c01ef4ebd62c" containerName="istio-proxy" Oct 31 19:40:44.164134 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.164002 4910 cpu_manager.go:414] "containerMap: removing container" podUID="638cecc0-b837-4dfb-9651-793b6f2986b2" containerName="istio-proxy" Oct 31 19:40:44.164134 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.164009 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="638cecc0-b837-4dfb-9651-793b6f2986b2" containerName="istio-proxy" Oct 31 19:40:44.164134 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.164050 4910 cpu_manager.go:414] "containerMap: removing container" podUID="20ac06c4-37b4-4da9-8c00-0c91d0d4143b" containerName="revoke-keys" Oct 31 19:40:44.164134 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.164058 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20ac06c4-37b4-4da9-8c00-0c91d0d4143b" containerName="revoke-keys" Oct 31 19:40:44.164134 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.164078 4910 cpu_manager.go:414] "containerMap: removing container" podUID="505b718c-f2ab-4f6d-892a-ed614bc2335d" containerName="maas-api" Oct 31 19:40:44.164134 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.164085 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="505b718c-f2ab-4f6d-892a-ed614bc2335d" containerName="maas-api" Oct 31 19:40:44.164534 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.164284 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="1540ff89-ac68-464d-b39f-c01ef4ebd62c" containerName="istio-proxy" Oct 31 19:40:44.164534 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.164302 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="638cecc0-b837-4dfb-9651-793b6f2986b2" containerName="istio-proxy" Oct 31 19:40:44.164534 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.164322 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="20ac06c4-37b4-4da9-8c00-0c91d0d4143b" containerName="revoke-keys" Oct 31 19:40:44.164534 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.164332 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="505b718c-f2ab-4f6d-892a-ed614bc2335d" containerName="maas-api" Oct 31 19:40:44.164534 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.164344 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="71387fb8-12d0-4620-a56c-7b4cd6ba554e" containerName="istio-proxy" Oct 31 19:40:44.168281 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.168258 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq" Oct 31 19:40:44.175090 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.175064 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-1c0c1d\"" Oct 31 19:40:44.177503 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.177478 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq"] Oct 31 19:40:44.359530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.359484 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc611929-0e53-4f9f-88f7-89beb3685be8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq\" (UID: \"cc611929-0e53-4f9f-88f7-89beb3685be8\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq" Oct 31 19:40:44.359742 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.359648 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8878\" (UniqueName: \"kubernetes.io/projected/cc611929-0e53-4f9f-88f7-89beb3685be8-kube-api-access-b8878\") pod \"payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq\" (UID: \"cc611929-0e53-4f9f-88f7-89beb3685be8\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq" Oct 31 19:40:44.467993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.467551 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc611929-0e53-4f9f-88f7-89beb3685be8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq\" (UID: \"cc611929-0e53-4f9f-88f7-89beb3685be8\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq" Oct 31 19:40:44.467993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.467724 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8878\" (UniqueName: \"kubernetes.io/projected/cc611929-0e53-4f9f-88f7-89beb3685be8-kube-api-access-b8878\") pod \"payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq\" (UID: \"cc611929-0e53-4f9f-88f7-89beb3685be8\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq" Oct 31 19:40:44.471935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.469677 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc611929-0e53-4f9f-88f7-89beb3685be8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq\" (UID: \"cc611929-0e53-4f9f-88f7-89beb3685be8\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq" Oct 31 19:40:44.518936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.518722 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8878\" (UniqueName: \"kubernetes.io/projected/cc611929-0e53-4f9f-88f7-89beb3685be8-kube-api-access-b8878\") pod \"payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq\" (UID: \"cc611929-0e53-4f9f-88f7-89beb3685be8\") " pod="openshift-ingress/payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq" Oct 31 19:40:44.790043 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:44.789958 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq" Oct 31 19:40:45.075351 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:40:45.075312 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc611929_0e53_4f9f_88f7_89beb3685be8.slice/crio-bfefe43595f1a172d285a8790fe9887c5b172bef57d6bb7ef6c76367e4ed166f WatchSource:0}: Error finding container bfefe43595f1a172d285a8790fe9887c5b172bef57d6bb7ef6c76367e4ed166f: Status 404 returned error can't find the container with id bfefe43595f1a172d285a8790fe9887c5b172bef57d6bb7ef6c76367e4ed166f Oct 31 19:40:45.077047 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:45.077024 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq"] Oct 31 19:40:45.173964 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:45.173820 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq" event={"ID":"cc611929-0e53-4f9f-88f7-89beb3685be8","Type":"ContainerStarted","Data":"bfefe43595f1a172d285a8790fe9887c5b172bef57d6bb7ef6c76367e4ed166f"} Oct 31 19:40:46.184738 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:46.184693 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq" event={"ID":"cc611929-0e53-4f9f-88f7-89beb3685be8","Type":"ContainerStarted","Data":"737394ed8be0a0df338a13655f4c57fc175ef90237b15e2147ff9eb01b136221"} Oct 31 19:40:46.185268 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:46.184775 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq" Oct 31 19:40:46.215742 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:46.215691 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq" podStartSLOduration=2.215677652 podStartE2EDuration="2.215677652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:46.212449909 +0000 UTC m=+1280.575270282" watchObservedRunningTime="2026-10-31 19:40:46.215677652 +0000 UTC m=+1280.578498023" Oct 31 19:40:47.713009 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.712977 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272"] Oct 31 19:40:47.717365 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.717340 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.736301 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.735806 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf"] Oct 31 19:40:47.742847 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.742823 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272"] Oct 31 19:40:47.743032 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.742940 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.776742 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.776699 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.776941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.776823 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.777038 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.776942 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wslr\" (UniqueName: \"kubernetes.io/projected/6f712963-3098-4e5d-b62d-df829fd9a55b-kube-api-access-5wslr\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.777101 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.777034 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.777208 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.777144 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.777271 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.777247 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.777338 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.777289 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.777396 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.777346 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.777475 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.777442 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f712963-3098-4e5d-b62d-df829fd9a55b-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.787089 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.787062 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf"] Oct 31 19:40:47.879058 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.879021 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.879242 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.879075 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.879242 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.879135 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqccl\" (UniqueName: \"kubernetes.io/projected/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-kube-api-access-wqccl\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.879242 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.879193 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.879242 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.879215 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.879407 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.879263 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.879407 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.879292 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.879407 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.879314 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.879407 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.879366 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.879615 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.879454 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f712963-3098-4e5d-b62d-df829fd9a55b-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.879615 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.879497 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.879615 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.879522 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.879615 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.879551 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.879823 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.879643 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.879823 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.879775 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.879823 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.879810 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.879978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.879842 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.879978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.879896 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.880058 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.879988 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wslr\" (UniqueName: \"kubernetes.io/projected/6f712963-3098-4e5d-b62d-df829fd9a55b-kube-api-access-5wslr\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.880058 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.880016 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.880151 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.880130 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f712963-3098-4e5d-b62d-df829fd9a55b-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.880406 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.880389 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.880560 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.880544 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.882280 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.882255 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.882576 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.882553 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.892229 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.892199 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.892328 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.892298 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wslr\" (UniqueName: \"kubernetes.io/projected/6f712963-3098-4e5d-b62d-df829fd9a55b-kube-api-access-5wslr\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-sw272\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:47.981641 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.981545 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.981641 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.981609 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqccl\" (UniqueName: \"kubernetes.io/projected/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-kube-api-access-wqccl\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.981874 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.981681 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.981874 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.981734 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.981874 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.981782 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.981874 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.981830 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.982115 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.982068 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.982171 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.982124 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.982255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.982242 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.982325 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.982260 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.982382 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.982358 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.982478 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.982451 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.982610 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.982491 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.982610 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.982566 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.984076 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.984052 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.984448 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.984429 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.995907 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.995884 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqccl\" (UniqueName: \"kubernetes.io/projected/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-kube-api-access-wqccl\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:47.999598 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:47.999575 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:48.040075 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:48.040045 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:48.099934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:48.095240 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:48.426579 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:40:48.426543 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f712963_3098_4e5d_b62d_df829fd9a55b.slice/crio-30af2674b6781fc938b70edb42eb864d5e9fb869b2ff5b97013611353c6a75a3 WatchSource:0}: Error finding container 30af2674b6781fc938b70edb42eb864d5e9fb869b2ff5b97013611353c6a75a3: Status 404 returned error can't find the container with id 30af2674b6781fc938b70edb42eb864d5e9fb869b2ff5b97013611353c6a75a3 Oct 31 19:40:48.426775 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:48.426753 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272"] Oct 31 19:40:48.430397 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:48.430223 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:40:48.430552 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:48.430483 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:40:48.430552 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:48.430533 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:40:48.444639 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:40:48.444597 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7e0301_710e_4fcb_8e00_6bc3ddccfedf.slice/crio-dd1f14f4badf3c1d7312fcbe2271c562de2fc5506f03844de27d9056e51db4fa WatchSource:0}: Error finding container dd1f14f4badf3c1d7312fcbe2271c562de2fc5506f03844de27d9056e51db4fa: Status 404 returned error can't find the container with id dd1f14f4badf3c1d7312fcbe2271c562de2fc5506f03844de27d9056e51db4fa Oct 31 19:40:48.447590 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:48.447563 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf"] Oct 31 19:40:48.447701 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:48.447637 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:40:48.447759 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:48.447701 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:40:48.447759 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:48.447736 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:40:49.256870 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:49.256825 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" event={"ID":"6f712963-3098-4e5d-b62d-df829fd9a55b","Type":"ContainerStarted","Data":"98d6b358e9ba3ea26ccc7695e8377610d0e4a0468e232e9aafc77855b4cdebe6"} Oct 31 19:40:49.259677 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:49.258497 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" event={"ID":"6f712963-3098-4e5d-b62d-df829fd9a55b","Type":"ContainerStarted","Data":"30af2674b6781fc938b70edb42eb864d5e9fb869b2ff5b97013611353c6a75a3"} Oct 31 19:40:49.259677 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:49.260503 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" event={"ID":"df7e0301-710e-4fcb-8e00-6bc3ddccfedf","Type":"ContainerStarted","Data":"ef3b9adf524171a46ebda8465dbf13911e6ddb80ddfb0d6338e60f23060aa3f9"} Oct 31 19:40:49.259677 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:49.260545 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" event={"ID":"df7e0301-710e-4fcb-8e00-6bc3ddccfedf","Type":"ContainerStarted","Data":"dd1f14f4badf3c1d7312fcbe2271c562de2fc5506f03844de27d9056e51db4fa"} Oct 31 19:40:49.295015 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:49.294937 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" podStartSLOduration=2.294907112 podStartE2EDuration="2.294907112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:49.289049844 +0000 UTC m=+1283.651870219" watchObservedRunningTime="2026-10-31 19:40:49.294907112 +0000 UTC m=+1283.657727483" Oct 31 19:40:49.325215 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:49.325152 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" podStartSLOduration=2.325134369 podStartE2EDuration="2.325134369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:49.318424529 +0000 UTC m=+1283.681244913" watchObservedRunningTime="2026-10-31 19:40:49.325134369 +0000 UTC m=+1283.687954741" Oct 31 19:40:49.903945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:49.899344 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9"] Oct 31 19:40:49.903945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:49.902425 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" podUID="790bd78d-4525-4587-9b6f-30af5841a88a" containerName="istio-proxy" containerID="cri-o://046c42181792f3e97062e05ffee400dd07f1509f7754c952cadb1a8be35e1b9d" gracePeriod=30 Oct 31 19:40:50.052939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:50.049222 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:50.052939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:50.050545 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 19:40:50.052939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:50.050599 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" podUID="6f712963-3098-4e5d-b62d-df829fd9a55b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 19:40:50.125936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:50.097102 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:50.125936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:50.125056 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 19:40:50.125936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:50.125163 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" podUID="df7e0301-710e-4fcb-8e00-6bc3ddccfedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 19:40:51.059213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:51.059181 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 19:40:51.071701 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:51.071661 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" podUID="6f712963-3098-4e5d-b62d-df829fd9a55b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 19:40:51.112606 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:51.109529 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 19:40:51.112606 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:51.109590 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" podUID="df7e0301-710e-4fcb-8e00-6bc3ddccfedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 19:40:51.811119 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:51.811076 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:40:52.005934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:52.003028 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:40:52.063998 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:52.057490 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 19:40:52.063998 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:52.057553 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" podUID="6f712963-3098-4e5d-b62d-df829fd9a55b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 19:40:52.145067 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:52.145028 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 19:40:52.145365 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:52.145331 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" podUID="df7e0301-710e-4fcb-8e00-6bc3ddccfedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 19:40:53.083933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:53.043142 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 19:40:53.084628 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:53.084591 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" podUID="6f712963-3098-4e5d-b62d-df829fd9a55b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 19:40:53.125579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:53.125544 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 19:40:53.125859 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:53.125830 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" podUID="df7e0301-710e-4fcb-8e00-6bc3ddccfedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 19:40:54.061938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:54.061692 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:40:54.077214 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:54.066765 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:40:54.077214 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:54.074505 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 19:40:54.077214 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:54.074561 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" podUID="6f712963-3098-4e5d-b62d-df829fd9a55b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 19:40:54.096955 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:54.095821 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 19:40:54.096955 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:54.095878 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" podUID="df7e0301-710e-4fcb-8e00-6bc3ddccfedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 19:40:55.041762 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:55.041730 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 19:40:55.042028 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:55.042000 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" podUID="6f712963-3098-4e5d-b62d-df829fd9a55b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 19:40:55.096006 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:55.095963 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 19:40:55.096278 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:55.096232 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" podUID="df7e0301-710e-4fcb-8e00-6bc3ddccfedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 19:40:56.262011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.202319 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 19:40:56.262011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.202380 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" podUID="df7e0301-710e-4fcb-8e00-6bc3ddccfedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 19:40:56.262011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.207127 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 19:40:56.262011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.207173 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" podUID="6f712963-3098-4e5d-b62d-df829fd9a55b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 19:40:56.377756 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.377253 4910 generic.go:356] "Generic (PLEG): container finished" podID="790bd78d-4525-4587-9b6f-30af5841a88a" containerID="046c42181792f3e97062e05ffee400dd07f1509f7754c952cadb1a8be35e1b9d" exitCode=0 Oct 31 19:40:56.377756 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.377489 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" event={"ID":"790bd78d-4525-4587-9b6f-30af5841a88a","Type":"ContainerDied","Data":"046c42181792f3e97062e05ffee400dd07f1509f7754c952cadb1a8be35e1b9d"} Oct 31 19:40:56.377756 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.377548 4910 scope.go:122] "RemoveContainer" containerID="db127c0d1f3a7531ff62c648fe12abfcc64371e36d96e7f50f8bcfeadc1ff6a7" Oct 31 19:40:56.414682 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.414650 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:40:56.477037 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.476991 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:40:56.491449 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.491412 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-istio-data\") pod \"790bd78d-4525-4587-9b6f-30af5841a88a\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " Oct 31 19:40:56.491759 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.491735 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/790bd78d-4525-4587-9b6f-30af5841a88a-kube-api-access-vxt2p\" (UniqueName: \"kubernetes.io/projected/790bd78d-4525-4587-9b6f-30af5841a88a-kube-api-access-vxt2p\") pod \"790bd78d-4525-4587-9b6f-30af5841a88a\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " Oct 31 19:40:56.491988 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.491969 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/790bd78d-4525-4587-9b6f-30af5841a88a-istio-token\" (UniqueName: \"kubernetes.io/projected/790bd78d-4525-4587-9b6f-30af5841a88a-istio-token\") pod \"790bd78d-4525-4587-9b6f-30af5841a88a\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " Oct 31 19:40:56.492342 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.492320 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/790bd78d-4525-4587-9b6f-30af5841a88a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790bd78d-4525-4587-9b6f-30af5841a88a-istiod-ca-cert\") pod \"790bd78d-4525-4587-9b6f-30af5841a88a\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " Oct 31 19:40:56.492508 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.492491 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-istio-envoy\") pod \"790bd78d-4525-4587-9b6f-30af5841a88a\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " Oct 31 19:40:56.492624 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.492610 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-workload-socket\") pod \"790bd78d-4525-4587-9b6f-30af5841a88a\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " Oct 31 19:40:56.492764 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.492747 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/790bd78d-4525-4587-9b6f-30af5841a88a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790bd78d-4525-4587-9b6f-30af5841a88a-istio-podinfo\") pod \"790bd78d-4525-4587-9b6f-30af5841a88a\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " Oct 31 19:40:56.492893 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.492879 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-credential-socket\") pod \"790bd78d-4525-4587-9b6f-30af5841a88a\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " Oct 31 19:40:56.493056 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.493040 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-workload-certs\") pod \"790bd78d-4525-4587-9b6f-30af5841a88a\" (UID: \"790bd78d-4525-4587-9b6f-30af5841a88a\") " Oct 31 19:40:56.514951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.514863 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-workload-certs" pod "790bd78d-4525-4587-9b6f-30af5841a88a" (UID: "790bd78d-4525-4587-9b6f-30af5841a88a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:40:56.515095 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.515048 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-workload-socket" pod "790bd78d-4525-4587-9b6f-30af5841a88a" (UID: "790bd78d-4525-4587-9b6f-30af5841a88a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:40:56.515610 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.515585 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/790bd78d-4525-4587-9b6f-30af5841a88a-istiod-ca-cert" pod "790bd78d-4525-4587-9b6f-30af5841a88a" (UID: "790bd78d-4525-4587-9b6f-30af5841a88a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:40:56.521944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.516965 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-istio-data" pod "790bd78d-4525-4587-9b6f-30af5841a88a" (UID: "790bd78d-4525-4587-9b6f-30af5841a88a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:40:56.521944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.517802 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-credential-socket" pod "790bd78d-4525-4587-9b6f-30af5841a88a" (UID: "790bd78d-4525-4587-9b6f-30af5841a88a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:40:56.554826 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.554214 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:40:56.554826 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.554569 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790bd78d-4525-4587-9b6f-30af5841a88a-istio-token" pod "790bd78d-4525-4587-9b6f-30af5841a88a" (UID: "790bd78d-4525-4587-9b6f-30af5841a88a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:40:56.634760 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.634720 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/790bd78d-4525-4587-9b6f-30af5841a88a-istio-podinfo" pod "790bd78d-4525-4587-9b6f-30af5841a88a" (UID: "790bd78d-4525-4587-9b6f-30af5841a88a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:40:56.668106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.639597 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:56.668106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.639630 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/790bd78d-4525-4587-9b6f-30af5841a88a-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:56.668106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.639644 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:56.668106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.639659 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:56.668106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.639684 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:56.668106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.639697 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/790bd78d-4525-4587-9b6f-30af5841a88a-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:56.668106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.639711 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/790bd78d-4525-4587-9b6f-30af5841a88a-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:56.704279 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.704181 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:40:56.705751 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.705705 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790bd78d-4525-4587-9b6f-30af5841a88a-kube-api-access-vxt2p" pod "790bd78d-4525-4587-9b6f-30af5841a88a" (UID: "790bd78d-4525-4587-9b6f-30af5841a88a"). InnerVolumeSpecName "kube-api-access-vxt2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:40:56.708633 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.708605 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-istio-envoy" pod "790bd78d-4525-4587-9b6f-30af5841a88a" (UID: "790bd78d-4525-4587-9b6f-30af5841a88a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:40:56.782991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.764460 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/790bd78d-4525-4587-9b6f-30af5841a88a-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:56.794610 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.791249 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxt2p\" (UniqueName: \"kubernetes.io/projected/790bd78d-4525-4587-9b6f-30af5841a88a-kube-api-access-vxt2p\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:40:56.803175 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.801329 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 19:40:56.817354 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.814854 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:40:56.887937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:56.885148 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:40:57.041579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:57.041508 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 19:40:57.041820 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:57.041765 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" podUID="6f712963-3098-4e5d-b62d-df829fd9a55b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 19:40:57.097179 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:57.097146 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 19:40:57.097336 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:57.097194 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" podUID="df7e0301-710e-4fcb-8e00-6bc3ddccfedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 19:40:57.277145 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:57.277110 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq" Oct 31 19:40:57.573464 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:57.573050 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" Oct 31 19:40:57.582146 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:57.582104 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9" event={"ID":"790bd78d-4525-4587-9b6f-30af5841a88a","Type":"ContainerDied","Data":"9b3451259462fea753f446071eef6fa840e0f2832d86fd2f646d8944ce110df4"} Oct 31 19:40:57.582331 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:57.582251 4910 scope.go:122] "RemoveContainer" containerID="046c42181792f3e97062e05ffee400dd07f1509f7754c952cadb1a8be35e1b9d" Oct 31 19:40:57.648637 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:57.648355 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9"] Oct 31 19:40:57.656759 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:57.656721 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-db61066b-openshift-default-f9db4cc6f-b6sb9"] Oct 31 19:40:57.765768 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:57.765725 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="790bd78d-4525-4587-9b6f-30af5841a88a" path="/var/lib/kubelet/pods/790bd78d-4525-4587-9b6f-30af5841a88a/volumes" Oct 31 19:40:58.204995 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.204962 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:58.206299 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.206273 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:58.209682 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.209657 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:40:58.219747 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.219716 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:58.222939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.221015 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:58.226933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.226111 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:40:58.289225 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.289186 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd"] Oct 31 19:40:58.290957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.290062 4910 cpu_manager.go:414] "containerMap: removing container" podUID="790bd78d-4525-4587-9b6f-30af5841a88a" containerName="istio-proxy" Oct 31 19:40:58.290957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.290122 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="790bd78d-4525-4587-9b6f-30af5841a88a" containerName="istio-proxy" Oct 31 19:40:58.290957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.290150 4910 cpu_manager.go:414] "containerMap: removing container" podUID="790bd78d-4525-4587-9b6f-30af5841a88a" containerName="istio-proxy" Oct 31 19:40:58.290957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.290158 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="790bd78d-4525-4587-9b6f-30af5841a88a" containerName="istio-proxy" Oct 31 19:40:58.290957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.290392 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="790bd78d-4525-4587-9b6f-30af5841a88a" containerName="istio-proxy" Oct 31 19:40:58.290957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.290412 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="790bd78d-4525-4587-9b6f-30af5841a88a" containerName="istio-proxy" Oct 31 19:40:58.297357 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.297318 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.322962 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.318238 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-88eaf9-openshift-default-dockercfg-dk2qg\"" Oct 31 19:40:58.327295 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.325149 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-istio-data\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.327295 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.325288 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-workload-certs\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.327295 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.325425 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-workload-socket\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.327295 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.325464 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dv5k\" (UniqueName: \"kubernetes.io/projected/706169ef-5dc5-4533-86af-fd02de456ab0-kube-api-access-5dv5k\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.327295 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.325528 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/706169ef-5dc5-4533-86af-fd02de456ab0-istio-token\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.327295 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.325618 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-credential-socket\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.327295 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.325688 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/706169ef-5dc5-4533-86af-fd02de456ab0-istio-podinfo\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.327295 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.325837 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/706169ef-5dc5-4533-86af-fd02de456ab0-istiod-ca-cert\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.327295 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.325985 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-istio-envoy\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.328901 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.328872 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd"] Oct 31 19:40:58.428093 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.428047 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-istio-envoy\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.428272 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.428249 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-istio-data\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.428352 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.428336 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-workload-certs\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.428458 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.428417 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-workload-socket\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.428513 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.428481 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dv5k\" (UniqueName: \"kubernetes.io/projected/706169ef-5dc5-4533-86af-fd02de456ab0-kube-api-access-5dv5k\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.428562 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.428524 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/706169ef-5dc5-4533-86af-fd02de456ab0-istio-token\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.428611 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.428581 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-credential-socket\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.428657 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.428629 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/706169ef-5dc5-4533-86af-fd02de456ab0-istio-podinfo\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.428709 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.428668 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-workload-certs\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.428709 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.428686 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-istio-data\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.428804 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.428705 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/706169ef-5dc5-4533-86af-fd02de456ab0-istiod-ca-cert\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.429412 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.429096 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-credential-socket\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.429412 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.429175 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-workload-socket\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.429647 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.429620 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/706169ef-5dc5-4533-86af-fd02de456ab0-istiod-ca-cert\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.431308 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.431281 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-istio-envoy\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.431489 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.431469 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/706169ef-5dc5-4533-86af-fd02de456ab0-istio-podinfo\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.446354 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.446320 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/706169ef-5dc5-4533-86af-fd02de456ab0-istio-token\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.446487 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.446449 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dv5k\" (UniqueName: \"kubernetes.io/projected/706169ef-5dc5-4533-86af-fd02de456ab0-kube-api-access-5dv5k\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:58.646086 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:58.646039 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:40:59.352128 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:40:59.352095 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod706169ef_5dc5_4533_86af_fd02de456ab0.slice/crio-3bc4c4891a05c3274c252f9a3eaad890273a632dcf6a7ff687b28b922ed363f4 WatchSource:0}: Error finding container 3bc4c4891a05c3274c252f9a3eaad890273a632dcf6a7ff687b28b922ed363f4: Status 404 returned error can't find the container with id 3bc4c4891a05c3274c252f9a3eaad890273a632dcf6a7ff687b28b922ed363f4 Oct 31 19:40:59.354399 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:59.354264 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd"] Oct 31 19:40:59.356027 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:59.355989 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:40:59.356187 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:59.356077 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:40:59.356187 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:59.356121 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:40:59.680634 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:59.679621 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" event={"ID":"706169ef-5dc5-4533-86af-fd02de456ab0","Type":"ContainerStarted","Data":"c88f574089b8579100c712a86a00aa52ee8ae7808ac21f7ef64c261a98cbcff8"} Oct 31 19:40:59.680634 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:59.679692 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" event={"ID":"706169ef-5dc5-4533-86af-fd02de456ab0","Type":"ContainerStarted","Data":"3bc4c4891a05c3274c252f9a3eaad890273a632dcf6a7ff687b28b922ed363f4"} Oct 31 19:40:59.713060 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:40:59.712996 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" podStartSLOduration=1.712975824 podStartE2EDuration="1.712975824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:59.709311377 +0000 UTC m=+1294.072131754" watchObservedRunningTime="2026-10-31 19:40:59.712975824 +0000 UTC m=+1294.075796199" Oct 31 19:41:00.652629 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:00.652589 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:41:00.660399 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:00.660369 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:41:00.783684 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:00.780039 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:41:00.783684 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:00.783054 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:41:02.785978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.784935 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l"] Oct 31 19:41:02.795935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.791203 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:02.799092 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.799062 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l"] Oct 31 19:41:02.891588 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.891545 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:02.891588 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.891598 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:02.892161 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.892138 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:02.892640 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.892621 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:02.892733 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.892665 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:02.892846 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.892823 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84rt6\" (UniqueName: \"kubernetes.io/projected/69edbb82-5f76-4f2d-b70a-d99a96ea5817-kube-api-access-84rt6\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:02.892995 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.892975 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:02.893123 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.893104 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:02.893185 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.893165 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:02.994594 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.994557 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:02.995068 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.995035 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:02.995304 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.995125 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84rt6\" (UniqueName: \"kubernetes.io/projected/69edbb82-5f76-4f2d-b70a-d99a96ea5817-kube-api-access-84rt6\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:02.995304 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.995161 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:02.995440 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.995272 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:02.995440 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.995391 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:02.995542 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.995350 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:02.995693 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.995663 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:02.995752 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.995697 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:02.995752 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.995734 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:02.995847 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.995831 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:02.995968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.995899 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:02.996278 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.996240 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:02.996386 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.996292 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:02.998448 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.998415 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:02.998556 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:02.998515 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:03.011336 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:03.011309 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:03.011584 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:03.011564 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84rt6\" (UniqueName: \"kubernetes.io/projected/69edbb82-5f76-4f2d-b70a-d99a96ea5817-kube-api-access-84rt6\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:03.154395 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:03.154354 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:03.883653 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:03.875699 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:03.883653 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:03.875964 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:03.883653 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:03.876234 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:04.006629 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:04.006515 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" event={"ID":"69edbb82-5f76-4f2d-b70a-d99a96ea5817","Type":"ContainerStarted","Data":"9cba9c3396bf67099758e5b2eb143034adb8fbe27e77c802a8ef3d26a33b2c2d"} Oct 31 19:41:04.006629 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:04.006575 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l"] Oct 31 19:41:05.017475 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:05.017435 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" event={"ID":"69edbb82-5f76-4f2d-b70a-d99a96ea5817","Type":"ContainerStarted","Data":"aff1762f981bda8ee76ed72e7e6766186f23f87cdce941d1fe7d573cc7951da6"} Oct 31 19:41:05.080595 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:05.080536 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" podStartSLOduration=3.0805143839999998 podStartE2EDuration="3.080514384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:05.077236484 +0000 UTC m=+1299.440056856" watchObservedRunningTime="2026-10-31 19:41:05.080514384 +0000 UTC m=+1299.443334759" Oct 31 19:41:05.158932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:05.158592 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:05.158932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:05.158777 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 31 19:41:05.158932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:05.158834 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" podUID="69edbb82-5f76-4f2d-b70a-d99a96ea5817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 31 19:41:06.158985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:06.155268 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 31 19:41:06.158985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:06.155326 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" podUID="69edbb82-5f76-4f2d-b70a-d99a96ea5817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 31 19:41:07.157695 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:07.157250 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 31 19:41:07.157695 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:07.157314 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" podUID="69edbb82-5f76-4f2d-b70a-d99a96ea5817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 31 19:41:08.161940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:08.158349 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 31 19:41:08.161940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:08.158434 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" podUID="69edbb82-5f76-4f2d-b70a-d99a96ea5817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 31 19:41:09.177203 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:09.177163 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 31 19:41:09.177818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:09.177770 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" podUID="69edbb82-5f76-4f2d-b70a-d99a96ea5817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 31 19:41:10.189317 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:10.189281 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 31 19:41:10.189954 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:10.189890 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" podUID="69edbb82-5f76-4f2d-b70a-d99a96ea5817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 31 19:41:11.158935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:11.158292 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 31 19:41:11.158935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:11.158354 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" podUID="69edbb82-5f76-4f2d-b70a-d99a96ea5817" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 31 19:41:11.835458 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:11.824101 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:11.998952 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:11.998148 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:11.999316 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:11.999284 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:41:12.597343 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:12.597309 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:12.598081 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:12.598040 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:12.599717 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:12.599691 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:12.753202 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:41:12.753158 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:41:13.962550 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:13.962506 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk"] Oct 31 19:41:13.968297 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:13.968270 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk" Oct 31 19:41:13.981068 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:13.980954 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-f5c1ab-serving-cert\"" Oct 31 19:41:13.984847 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:13.984818 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk"] Oct 31 19:41:14.105496 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.104405 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx4qg\" (UniqueName: \"kubernetes.io/projected/27300f49-081d-43d0-b6c5-4d6b55f074ad-kube-api-access-sx4qg\") pod \"maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk\" (UID: \"27300f49-081d-43d0-b6c5-4d6b55f074ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk" Oct 31 19:41:14.105496 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.104532 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27300f49-081d-43d0-b6c5-4d6b55f074ad-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk\" (UID: \"27300f49-081d-43d0-b6c5-4d6b55f074ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk" Oct 31 19:41:14.105496 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.104690 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27300f49-081d-43d0-b6c5-4d6b55f074ad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk\" (UID: \"27300f49-081d-43d0-b6c5-4d6b55f074ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk" Oct 31 19:41:14.105496 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.104729 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27300f49-081d-43d0-b6c5-4d6b55f074ad-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk\" (UID: \"27300f49-081d-43d0-b6c5-4d6b55f074ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk" Oct 31 19:41:14.215550 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.215458 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27300f49-081d-43d0-b6c5-4d6b55f074ad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk\" (UID: \"27300f49-081d-43d0-b6c5-4d6b55f074ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk" Oct 31 19:41:14.215550 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.215511 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27300f49-081d-43d0-b6c5-4d6b55f074ad-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk\" (UID: \"27300f49-081d-43d0-b6c5-4d6b55f074ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk" Oct 31 19:41:14.215787 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.215763 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx4qg\" (UniqueName: \"kubernetes.io/projected/27300f49-081d-43d0-b6c5-4d6b55f074ad-kube-api-access-sx4qg\") pod \"maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk\" (UID: \"27300f49-081d-43d0-b6c5-4d6b55f074ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk" Oct 31 19:41:14.215858 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.215843 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27300f49-081d-43d0-b6c5-4d6b55f074ad-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk\" (UID: \"27300f49-081d-43d0-b6c5-4d6b55f074ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk" Oct 31 19:41:14.217145 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.217107 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27300f49-081d-43d0-b6c5-4d6b55f074ad-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk\" (UID: \"27300f49-081d-43d0-b6c5-4d6b55f074ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk" Oct 31 19:41:14.219106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.219036 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27300f49-081d-43d0-b6c5-4d6b55f074ad-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk\" (UID: \"27300f49-081d-43d0-b6c5-4d6b55f074ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk" Oct 31 19:41:14.225016 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.224962 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27300f49-081d-43d0-b6c5-4d6b55f074ad-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk\" (UID: \"27300f49-081d-43d0-b6c5-4d6b55f074ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk" Oct 31 19:41:14.233257 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.233228 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx4qg\" (UniqueName: \"kubernetes.io/projected/27300f49-081d-43d0-b6c5-4d6b55f074ad-kube-api-access-sx4qg\") pod \"maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk\" (UID: \"27300f49-081d-43d0-b6c5-4d6b55f074ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk" Oct 31 19:41:14.288591 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.288552 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk" Oct 31 19:41:14.789753 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.771265 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt"] Oct 31 19:41:14.825582 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.825528 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt"] Oct 31 19:41:14.825776 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.825657 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:14.915256 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.915223 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-fd1c91-openshift-default-dockercfg-hm6l2\"" Oct 31 19:41:14.925674 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.925221 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkrfn\" (UniqueName: \"kubernetes.io/projected/69515b0c-d32a-40b2-baf8-995554f2a5b9-kube-api-access-rkrfn\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:14.925674 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.925309 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-workload-certs\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:14.925674 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.925394 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-workload-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:14.925674 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.925480 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-podinfo\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:14.925674 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.925512 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-token\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:14.926095 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.925804 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-credential-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:14.926095 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.925866 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-data\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:14.926095 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.925972 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69515b0c-d32a-40b2-baf8-995554f2a5b9-istiod-ca-cert\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:14.926095 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:14.926001 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-envoy\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:15.029668 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.028234 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-credential-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:15.029668 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.028308 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-data\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:15.029668 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.028391 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69515b0c-d32a-40b2-baf8-995554f2a5b9-istiod-ca-cert\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:15.029668 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.028417 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-envoy\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:15.029668 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.028495 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkrfn\" (UniqueName: \"kubernetes.io/projected/69515b0c-d32a-40b2-baf8-995554f2a5b9-kube-api-access-rkrfn\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:15.029668 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.028543 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-workload-certs\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:15.029668 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.028603 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-workload-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:15.029668 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.028665 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-podinfo\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:15.029668 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.028687 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-token\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:15.029668 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.029380 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-credential-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:15.029668 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.029554 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-data\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:15.030714 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.029981 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69515b0c-d32a-40b2-baf8-995554f2a5b9-istiod-ca-cert\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:15.034992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.034966 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-workload-certs\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:15.036079 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.035409 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-workload-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:15.037551 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:41:15.037528 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27300f49_081d_43d0_b6c5_4d6b55f074ad.slice/crio-17a5a8fc3258694190dbfb6ac66d3117aa59a89d90587b9f233053263fda63e3 WatchSource:0}: Error finding container 17a5a8fc3258694190dbfb6ac66d3117aa59a89d90587b9f233053263fda63e3: Status 404 returned error can't find the container with id 17a5a8fc3258694190dbfb6ac66d3117aa59a89d90587b9f233053263fda63e3 Oct 31 19:41:15.038087 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.037982 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk"] Oct 31 19:41:15.040158 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.040096 4910 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 19:41:15.046512 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.046481 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-envoy\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:15.050503 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.050441 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-podinfo\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:15.095118 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.095083 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkrfn\" (UniqueName: \"kubernetes.io/projected/69515b0c-d32a-40b2-baf8-995554f2a5b9-kube-api-access-rkrfn\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:15.095299 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.095214 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-token\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:15.198935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.170245 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:15.198935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.190731 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8"] Oct 31 19:41:15.200146 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.200116 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.271933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.266826 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8"] Oct 31 19:41:15.274966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.274871 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh"] Oct 31 19:41:15.283000 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.281489 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.305048 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.304963 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh"] Oct 31 19:41:15.379530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.375489 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.379530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.375541 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.379530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.375580 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.379530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.375613 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.379530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.375807 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.379530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.375895 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts7s7\" (UniqueName: \"kubernetes.io/projected/465829b5-8a02-4035-b06d-a283ba96ae0f-kube-api-access-ts7s7\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.379530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.375975 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/465829b5-8a02-4035-b06d-a283ba96ae0f-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.379530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.376018 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.379530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.376069 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.379530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.376105 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.379530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.376145 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.379530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.376167 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.379530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.376196 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.379530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.376235 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.379530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.376282 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42lmz\" (UniqueName: \"kubernetes.io/projected/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-kube-api-access-42lmz\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.379530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.376336 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.379530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.376405 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.379530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.376443 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.477788 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.477760 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.478227 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.477817 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.478227 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.477849 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.478227 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.477896 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.478227 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.477940 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.478227 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.477972 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.478227 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.478007 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.478486 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.478239 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.478486 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.478305 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.478486 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.478317 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.478486 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.478336 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts7s7\" (UniqueName: \"kubernetes.io/projected/465829b5-8a02-4035-b06d-a283ba96ae0f-kube-api-access-ts7s7\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.478486 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.478305 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.478486 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.478368 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/465829b5-8a02-4035-b06d-a283ba96ae0f-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.478486 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.478413 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.478486 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.478465 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.478819 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.478499 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.478819 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.478544 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.478819 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.478568 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.478819 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.478598 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.478819 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.478627 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.478819 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.478671 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42lmz\" (UniqueName: \"kubernetes.io/projected/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-kube-api-access-42lmz\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.480045 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.479343 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.480045 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.479431 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.480045 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.479465 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.480045 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.479704 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.480045 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.479839 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.481042 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.480997 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/465829b5-8a02-4035-b06d-a283ba96ae0f-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.481148 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.481106 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.481148 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.481121 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.481148 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.481127 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.482643 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.482622 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.482728 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.482713 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.490836 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.490808 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.491253 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.491230 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.491343 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.491258 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts7s7\" (UniqueName: \"kubernetes.io/projected/465829b5-8a02-4035-b06d-a283ba96ae0f-kube-api-access-ts7s7\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.492978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.492960 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42lmz\" (UniqueName: \"kubernetes.io/projected/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-kube-api-access-42lmz\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.550956 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.550878 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:15.592027 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:41:15.591997 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69515b0c_d32a_40b2_baf8_995554f2a5b9.slice/crio-ed6620e044e13c01bd0f20d8fd3e9d2629d5f0eadda07beac6392beefbce3b82 WatchSource:0}: Error finding container ed6620e044e13c01bd0f20d8fd3e9d2629d5f0eadda07beac6392beefbce3b82: Status 404 returned error can't find the container with id ed6620e044e13c01bd0f20d8fd3e9d2629d5f0eadda07beac6392beefbce3b82 Oct 31 19:41:15.596421 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.595803 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:15.596421 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.595848 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt"] Oct 31 19:41:15.596421 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.595899 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:15.596421 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.595961 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:15.623580 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.623327 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:15.677518 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.677480 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" event={"ID":"69515b0c-d32a-40b2-baf8-995554f2a5b9","Type":"ContainerStarted","Data":"ed6620e044e13c01bd0f20d8fd3e9d2629d5f0eadda07beac6392beefbce3b82"} Oct 31 19:41:15.684994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.684590 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk" event={"ID":"27300f49-081d-43d0-b6c5-4d6b55f074ad","Type":"ContainerStarted","Data":"d12f6f7db05e4c344e0fd92916175cc490d8966446630f330d0276e2de88147f"} Oct 31 19:41:15.684994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.684657 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk" event={"ID":"27300f49-081d-43d0-b6c5-4d6b55f074ad","Type":"ContainerStarted","Data":"17a5a8fc3258694190dbfb6ac66d3117aa59a89d90587b9f233053263fda63e3"} Oct 31 19:41:15.685281 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.685260 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk" Oct 31 19:41:15.714497 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.714438 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk" podStartSLOduration=2.300375565 podStartE2EDuration="2.714417895s" totalImagesPullingTime="414.04233ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:15.708007398 +0000 UTC m=+1310.070827771" watchObservedRunningTime="2026-10-31 19:41:15.714417895 +0000 UTC m=+1310.077238270" Oct 31 19:41:15.967749 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.966609 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:15.967749 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.966705 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:15.967749 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.966749 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:15.974753 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:15.973694 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8"] Oct 31 19:41:16.227759 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:16.227723 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh"] Oct 31 19:41:16.238287 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:16.238237 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:16.238451 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:16.238341 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:16.238451 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:16.238388 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:16.707392 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:16.706338 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" event={"ID":"465829b5-8a02-4035-b06d-a283ba96ae0f","Type":"ContainerStarted","Data":"3fc8bf7e4c936c02e8b32d083d2fbe8a5e90b1ca1a8810fd153183d2bb091b3e"} Oct 31 19:41:16.707392 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:16.706407 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" event={"ID":"465829b5-8a02-4035-b06d-a283ba96ae0f","Type":"ContainerStarted","Data":"6d41020e2c555694df431ea819edd34efc34b02d4f18d7ef530dc5aef10f0dee"} Oct 31 19:41:16.711677 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:16.711644 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" event={"ID":"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8","Type":"ContainerStarted","Data":"25cbe54f224369bb6112062e2f930c6be73fab0040c3ebf55c5ca0e56a147dbe"} Oct 31 19:41:16.711854 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:16.711701 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" event={"ID":"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8","Type":"ContainerStarted","Data":"0a2b202af9b0a2e79694445c882936fcb9e71c2861ca8f23730834fbbd213160"} Oct 31 19:41:16.715323 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:16.715283 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" event={"ID":"69515b0c-d32a-40b2-baf8-995554f2a5b9","Type":"ContainerStarted","Data":"8ec59dc90c2c484fed199e1a4a9204e47f52ebddd09cc8d7a7378d99759d04ae"} Oct 31 19:41:16.732634 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:16.732585 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" podStartSLOduration=1.732570062 podStartE2EDuration="1.732570062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:16.732112813 +0000 UTC m=+1311.094933186" watchObservedRunningTime="2026-10-31 19:41:16.732570062 +0000 UTC m=+1311.095390500" Oct 31 19:41:16.799722 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:16.799660 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" podStartSLOduration=2.799645668 podStartE2EDuration="2.799645668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:16.764705704 +0000 UTC m=+1311.127526078" watchObservedRunningTime="2026-10-31 19:41:16.799645668 +0000 UTC m=+1311.162466039" Oct 31 19:41:16.806120 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:16.806067 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" podStartSLOduration=1.806050114 podStartE2EDuration="1.806050114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:16.803447488 +0000 UTC m=+1311.166267945" watchObservedRunningTime="2026-10-31 19:41:16.806050114 +0000 UTC m=+1311.168870491" Oct 31 19:41:17.181842 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:17.181805 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:17.182239 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:17.182216 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 31 19:41:17.196075 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:17.196016 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" podUID="69515b0c-d32a-40b2-baf8-995554f2a5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 31 19:41:17.558039 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:17.556780 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:17.584354 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:17.583944 4910 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 31 19:41:17.584354 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:17.584057 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" podUID="5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 31 19:41:17.626149 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:17.626097 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:17.626345 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:17.626268 4910 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 19:41:17.626414 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:17.626340 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" podUID="465829b5-8a02-4035-b06d-a283ba96ae0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 19:41:18.171410 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:18.171371 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 31 19:41:18.171709 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:18.171678 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" podUID="69515b0c-d32a-40b2-baf8-995554f2a5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 31 19:41:18.199577 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:18.199031 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:18.585000 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:18.559306 4910 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 31 19:41:18.585000 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:18.559371 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" podUID="5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 31 19:41:18.760938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:18.747603 4910 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 19:41:18.788293 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:18.761867 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" podUID="465829b5-8a02-4035-b06d-a283ba96ae0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 19:41:19.089313 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.076751 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:19.176121 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.171056 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq"] Oct 31 19:41:19.216934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.207163 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:19.245236 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.245197 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 31 19:41:19.245514 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.245484 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" podUID="69515b0c-d32a-40b2-baf8-995554f2a5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 31 19:41:19.246462 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.212784 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:19.254189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.254164 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:41:19.316938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.308546 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:19.366992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.347026 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-podinfo\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:19.366992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.347109 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-envoy\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:19.366992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.347230 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-workload-certs\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:19.366992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.347261 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istiod-ca-cert\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:19.366992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.347294 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-data\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:19.366992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.347571 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-token\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:19.366992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.347638 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5nxn\" (UniqueName: \"kubernetes.io/projected/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-kube-api-access-n5nxn\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:19.366992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.347796 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-credential-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:19.366992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.347903 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-workload-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:19.485934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.481534 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:19.495942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.489704 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-token\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:19.495942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.489788 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5nxn\" (UniqueName: \"kubernetes.io/projected/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-kube-api-access-n5nxn\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:19.495942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.489880 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-credential-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:19.495942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.489968 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-workload-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:19.495942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.490068 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-podinfo\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:19.495942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.490133 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-envoy\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:19.495942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.490221 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-workload-certs\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:19.495942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.490252 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istiod-ca-cert\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:19.495942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.490284 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-data\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:19.495942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.490800 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-data\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:19.495942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.491666 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-credential-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:19.495942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.491970 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-workload-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:19.542935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.499239 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-workload-certs\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:19.542935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.500238 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istiod-ca-cert\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:19.692180 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.692056 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq"] Oct 31 19:41:19.693262 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.693233 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-envoy\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:19.693501 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.693481 4910 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 19:41:19.693647 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.693623 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" podUID="465829b5-8a02-4035-b06d-a283ba96ae0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 19:41:19.765282 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:19.697509 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:20.107430 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:20.107382 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5nxn\" (UniqueName: \"kubernetes.io/projected/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-kube-api-access-n5nxn\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:20.107819 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:20.107778 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:20.179473 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:20.179413 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 19:41:20.181627 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:20.181597 4910 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 31 19:41:20.181825 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:20.181800 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" podUID="5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 31 19:41:20.193251 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:20.193214 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-token\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:20.231979 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:20.231942 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 31 19:41:20.232259 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:20.232229 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" podUID="69515b0c-d32a-40b2-baf8-995554f2a5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 31 19:41:20.333932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:20.333861 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:20.347382 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:20.109936 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-podinfo\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:20.347382 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:20.344656 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:41:20.347382 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:20.344887 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:41:20.347382 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:20.346844 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:20.388993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:20.369606 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:41:20.494170 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:20.494123 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:20.514998 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:20.503740 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:20.642474 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:20.642381 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:20.706960 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:20.699697 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:41:20.720987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:20.711095 4910 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 31 19:41:20.720987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:20.711157 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" podUID="5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 31 19:41:20.720987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:20.711574 4910 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 19:41:20.720987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:20.711612 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" podUID="465829b5-8a02-4035-b06d-a283ba96ae0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 19:41:20.907855 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:20.903052 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:41:21.162994 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:41:21.158403 4910 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.403s" Oct 31 19:41:21.162994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.158552 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:41:21.162994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.158573 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf"] Oct 31 19:41:21.162994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.158608 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h"] Oct 31 19:41:21.162994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.158659 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:41:21.162994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.158676 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx"] Oct 31 19:41:21.162994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.158705 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr"] Oct 31 19:41:21.162994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.158758 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:41:21.162994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.158792 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:41:21.162994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.158821 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:41:21.163607 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.163579 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" podUID="005cc55c-8e13-47d1-aa18-ddb1fb79432c" containerName="istio-proxy" containerID="cri-o://a0ec8840783025046c13d0a7b093c07e4ff6fc0d2810bf63476f2685c21eaacf" gracePeriod=30 Oct 31 19:41:21.194178 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.192346 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 31 19:41:21.194178 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.192393 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" podUID="69515b0c-d32a-40b2-baf8-995554f2a5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 31 19:41:21.194178 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.193688 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" podUID="09020e9b-b4e8-4cbc-946e-73f3f7e8fc94" containerName="istio-proxy" containerID="cri-o://ad1a9a9e89e144c56863eac13e70e8b50eff089d96858192c0bc62c97d2bc165" gracePeriod=30 Oct 31 19:41:21.194178 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.193897 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" podUID="f9561493-df06-4f24-878e-f5d1255ac41b" containerName="istio-proxy" containerID="cri-o://9d1c823c6925f1ab683f3f5a436323f906f59f17d4115922a296ef3de285a0e0" gracePeriod=30 Oct 31 19:41:21.194178 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.194060 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" podUID="6ab4bc46-8c70-4f5a-8e0d-f852190f8c16" containerName="istio-proxy" containerID="cri-o://42d858a26dd2495bb3c2775ccee03fa7d591627c25db84bd9eef77f8831fb972" gracePeriod=30 Oct 31 19:41:21.229072 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.228298 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:21.414163 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.386212 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:21.414163 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.388867 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:21.414163 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.389231 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:41:21.414163 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.389481 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:41:21.414163 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.389767 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:41:21.414163 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.403202 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:21.607387 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.580162 4910 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 31 19:41:21.607387 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.580234 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" podUID="5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 31 19:41:21.644935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.627302 4910 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 19:41:21.644935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.627358 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" podUID="465829b5-8a02-4035-b06d-a283ba96ae0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 19:41:21.711545 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.711450 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:21.896235 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.896181 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:41:21.911055 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:21.911005 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:41:22.158172 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:22.158119 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:22.277730 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:22.177376 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:22.277730 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:22.180517 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:22.277730 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:22.226734 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:22.388887 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:22.294453 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 31 19:41:22.388887 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:22.294508 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" podUID="69515b0c-d32a-40b2-baf8-995554f2a5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 31 19:41:22.388887 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:22.316633 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:41:22.388887 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:22.335104 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:22.439348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:22.439310 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:41:22.440799 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:22.440766 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:41:22.631983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:22.546443 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:22.631983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:22.604326 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:41:22.631983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:22.604407 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:41:22.631983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:22.616507 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:41:22.689355 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:22.663278 4910 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 31 19:41:22.689355 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:22.663346 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" podUID="5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 31 19:41:22.715404 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:22.715345 4910 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 19:41:22.716753 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:22.716718 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" podUID="465829b5-8a02-4035-b06d-a283ba96ae0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 19:41:22.759934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:22.746141 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:22.824048 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:22.819168 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk" Oct 31 19:41:22.824048 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:22.824005 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:41:22.826272 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:22.826240 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:22.907997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:22.900022 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:22.916031 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:22.915998 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:41:23.004932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.001631 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:41:23.047964 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.047906 4910 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:41:23.237996 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.104025 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" podUID="507b2fb7-a22a-4cac-9d4a-23199f7fcbff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:23.237996 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.196710 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:23.237996 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.197148 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:41:23.237996 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.208242 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:41:23.237996 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.208315 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:41:23.237996 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.210298 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 31 19:41:23.237996 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.210344 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" podUID="69515b0c-d32a-40b2-baf8-995554f2a5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 31 19:41:23.267092 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.267048 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:41:23.363183 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.267585 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:41:23.379443 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:22.651285 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 19:41:23.403935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.389034 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:41:23.616101 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.489477 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:23.645415 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.470277 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:23.659037 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.592361 4910 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 31 19:41:23.659037 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.647748 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" podUID="5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 31 19:41:23.659037 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.470185 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:23.773933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.773542 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:23.773933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.773748 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:23.773933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.773851 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:23.849860 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.837123 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:23.849860 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.837703 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:41:23.911327 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.909548 4910 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 19:41:23.911327 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.909657 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" podUID="465829b5-8a02-4035-b06d-a283ba96ae0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 19:41:23.968933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.955492 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:41:23.970898 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.970860 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:41:23.999921 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:23.999881 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:41:24.102090 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:24.102024 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:24.285034 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:24.275628 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:24.404254 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:24.404198 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:41:24.476946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:24.461566 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:24.677559 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:24.477749 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:24.695954 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:24.477793 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded" Oct 31 19:41:24.695954 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:24.525717 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:24.789500 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:24.789445 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:25.314265 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:25.314205 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:25.609258 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:24.867294 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:41:25.609640 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:24.867756 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:41:25.610066 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:24.867794 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:41:25.610271 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:24.918630 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 31 19:41:25.610430 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:25.610403 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" podUID="69515b0c-d32a-40b2-baf8-995554f2a5b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 31 19:41:25.610561 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:25.144258 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:25.610703 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:25.441500 4910 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Oct 31 19:41:25.610799 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:25.610784 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" podUID="5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Oct 31 19:41:25.610971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:25.308491 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:25.615031 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:24.865059 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:25.615212 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:25.320591 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:41:25.615332 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:25.463829 4910 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 19:41:25.615421 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:25.615406 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" podUID="465829b5-8a02-4035-b06d-a283ba96ae0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 19:41:25.615499 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:25.612159 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:25.615618 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:25.360729 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:41:25.615740 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:25.612284 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": context deadline exceeded" Oct 31 19:41:25.686538 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:25.678446 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:25.795793 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:25.788932 4910 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 31 19:41:25.795793 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:25.788988 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" podUID="465829b5-8a02-4035-b06d-a283ba96ae0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 31 19:41:26.004259 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:41:26.004191 4910 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.864s" Oct 31 19:41:26.346985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.341239 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" event={"ID":"00f5bd2c-c4f2-4854-8169-dd65ad5d301d","Type":"ContainerStarted","Data":"1a99dd743d0ec534887b3ff879094ffc5874eaf5d0c748cf93b0c4a916e59cea"} Oct 31 19:41:26.346985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.341358 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:26.346985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.341394 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4"] Oct 31 19:41:26.346985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.341418 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d4b21c-htbz4"] Oct 31 19:41:26.346985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.341465 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:41:26.346985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.341486 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq"] Oct 31 19:41:26.346985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.341544 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:41:26.346985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.341578 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:41:26.346985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.341610 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:41:26.346985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.341640 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:41:26.346985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.341672 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:41:26.346985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.341700 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:41:26.346985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.341734 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:41:26.346985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.341766 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:41:26.346985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.341797 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:41:26.346985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.341828 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:41:26.346985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.341926 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:41:26.346985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.341970 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:41:26.349946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.349900 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:26.351561 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.351540 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:41:26.659662 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.659567 4910 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:41:26.659662 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.659622 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" podUID="5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:26.678209 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.678092 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf"] Oct 31 19:41:26.678407 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.678222 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:26.684304 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.679859 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" podUID="df7e0301-710e-4fcb-8e00-6bc3ddccfedf" containerName="istio-proxy" containerID="cri-o://ef3b9adf524171a46ebda8465dbf13911e6ddb80ddfb0d6338e60f23060aa3f9" gracePeriod=30 Oct 31 19:41:26.684304 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.680367 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:26.684304 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.681650 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:26.691014 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.688769 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" event={"ID":"00f5bd2c-c4f2-4854-8169-dd65ad5d301d","Type":"ContainerStarted","Data":"21f020cabaea277be7fcf2cdd96320e702873f241f117e4ee74ccddc0ca4fc4f"} Oct 31 19:41:26.694619 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.692235 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:26.695015 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.694902 4910 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 31 19:41:26.695015 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.694977 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" podUID="00f5bd2c-c4f2-4854-8169-dd65ad5d301d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 31 19:41:26.707350 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.695587 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272"] Oct 31 19:41:26.708815 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.708067 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" podUID="6f712963-3098-4e5d-b62d-df829fd9a55b" containerName="istio-proxy" containerID="cri-o://98d6b358e9ba3ea26ccc7695e8377610d0e4a0468e232e9aafc77855b4cdebe6" gracePeriod=30 Oct 31 19:41:26.769164 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.769038 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw"] Oct 31 19:41:26.769360 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.769267 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" podUID="26ae8974-942b-4bd3-9ca9-f047209ab0fa" containerName="istio-proxy" containerID="cri-o://d5ca693c42416ee872e735614fbde751da9997d42c679a8f1207cce9f047313e" gracePeriod=30 Oct 31 19:41:26.917987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.911078 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l"] Oct 31 19:41:26.917987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:26.911537 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" podUID="69edbb82-5f76-4f2d-b70a-d99a96ea5817" containerName="istio-proxy" containerID="cri-o://aff1762f981bda8ee76ed72e7e6766186f23f87cdce941d1fe7d573cc7951da6" gracePeriod=30 Oct 31 19:41:27.228871 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:27.212114 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:27.228871 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:27.213601 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:27.228871 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:27.228299 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:27.457741 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:27.457075 4910 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 31 19:41:27.457741 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:27.457130 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" podUID="f9561493-df06-4f24-878e-f5d1255ac41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 31 19:41:27.671925 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:27.647306 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:41:27.681842 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:27.681086 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" podUID="69edbb82-5f76-4f2d-b70a-d99a96ea5817" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:41:27.807749 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:27.800205 4910 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 31 19:41:27.815823 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:27.815775 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" podUID="00f5bd2c-c4f2-4854-8169-dd65ad5d301d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 31 19:41:28.075486 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.075284 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:41:28.157995 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.156986 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" podStartSLOduration=10.156966431 podStartE2EDuration="10.156966431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:27.597581471 +0000 UTC m=+1321.960401853" watchObservedRunningTime="2026-10-31 19:41:28.156966431 +0000 UTC m=+1322.519786804" Oct 31 19:41:28.172410 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.172371 4910 generic.go:356] "Generic (PLEG): container finished" podID="005cc55c-8e13-47d1-aa18-ddb1fb79432c" containerID="a0ec8840783025046c13d0a7b093c07e4ff6fc0d2810bf63476f2685c21eaacf" exitCode=0 Oct 31 19:41:28.172750 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.172725 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" Oct 31 19:41:28.175735 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.175702 4910 generic.go:356] "Generic (PLEG): container finished" podID="09020e9b-b4e8-4cbc-946e-73f3f7e8fc94" containerID="ad1a9a9e89e144c56863eac13e70e8b50eff089d96858192c0bc62c97d2bc165" exitCode=0 Oct 31 19:41:28.184104 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.184075 4910 generic.go:356] "Generic (PLEG): container finished" podID="6ab4bc46-8c70-4f5a-8e0d-f852190f8c16" containerID="42d858a26dd2495bb3c2775ccee03fa7d591627c25db84bd9eef77f8831fb972" exitCode=0 Oct 31 19:41:28.187033 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.187007 4910 generic.go:356] "Generic (PLEG): container finished" podID="f9561493-df06-4f24-878e-f5d1255ac41b" containerID="9d1c823c6925f1ab683f3f5a436323f906f59f17d4115922a296ef3de285a0e0" exitCode=0 Oct 31 19:41:28.223543 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.223494 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20ac06c4-37b4-4da9-8c00-0c91d0d4143b" path="/var/lib/kubelet/pods/20ac06c4-37b4-4da9-8c00-0c91d0d4143b/volumes" Oct 31 19:41:28.225223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.225193 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" event={"ID":"005cc55c-8e13-47d1-aa18-ddb1fb79432c","Type":"ContainerDied","Data":"a0ec8840783025046c13d0a7b093c07e4ff6fc0d2810bf63476f2685c21eaacf"} Oct 31 19:41:28.225610 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.225478 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf" event={"ID":"005cc55c-8e13-47d1-aa18-ddb1fb79432c","Type":"ContainerDied","Data":"d89da33412fe1b97965bd3ac6c77bb042e657a50b974746a384760d49ae18243"} Oct 31 19:41:28.225876 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.225856 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" event={"ID":"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94","Type":"ContainerDied","Data":"ad1a9a9e89e144c56863eac13e70e8b50eff089d96858192c0bc62c97d2bc165"} Oct 31 19:41:28.226068 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.226049 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" event={"ID":"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16","Type":"ContainerDied","Data":"42d858a26dd2495bb3c2775ccee03fa7d591627c25db84bd9eef77f8831fb972"} Oct 31 19:41:28.226290 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.226275 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" event={"ID":"f9561493-df06-4f24-878e-f5d1255ac41b","Type":"ContainerDied","Data":"9d1c823c6925f1ab683f3f5a436323f906f59f17d4115922a296ef3de285a0e0"} Oct 31 19:41:28.228498 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.227956 4910 scope.go:122] "RemoveContainer" containerID="a0ec8840783025046c13d0a7b093c07e4ff6fc0d2810bf63476f2685c21eaacf" Oct 31 19:41:28.244949 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.238685 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:41:28.244949 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.238762 4910 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:41:28.244949 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.238793 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" podUID="6f712963-3098-4e5d-b62d-df829fd9a55b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:41:28.244949 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.239102 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" podUID="df7e0301-710e-4fcb-8e00-6bc3ddccfedf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:41:28.276609 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.275829 4910 scope.go:122] "RemoveContainer" containerID="a0ec8840783025046c13d0a7b093c07e4ff6fc0d2810bf63476f2685c21eaacf" Oct 31 19:41:28.276609 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:41:28.276529 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0ec8840783025046c13d0a7b093c07e4ff6fc0d2810bf63476f2685c21eaacf\": container with ID starting with a0ec8840783025046c13d0a7b093c07e4ff6fc0d2810bf63476f2685c21eaacf not found: ID does not exist" containerID="a0ec8840783025046c13d0a7b093c07e4ff6fc0d2810bf63476f2685c21eaacf" Oct 31 19:41:28.276609 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.276566 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0ec8840783025046c13d0a7b093c07e4ff6fc0d2810bf63476f2685c21eaacf"} err="failed to get container status \"a0ec8840783025046c13d0a7b093c07e4ff6fc0d2810bf63476f2685c21eaacf\": rpc error: code = NotFound desc = could not find container \"a0ec8840783025046c13d0a7b093c07e4ff6fc0d2810bf63476f2685c21eaacf\": container with ID starting with a0ec8840783025046c13d0a7b093c07e4ff6fc0d2810bf63476f2685c21eaacf not found: ID does not exist" Oct 31 19:41:28.288088 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.287169 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:28.320393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.320315 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istiod-ca-cert" pod "005cc55c-8e13-47d1-aa18-ddb1fb79432c" (UID: "005cc55c-8e13-47d1-aa18-ddb1fb79432c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:41:28.334514 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.331131 4910 scope.go:122] "RemoveContainer" containerID="10c1225c10a8b30bdd673d74e3f70bab6afc2312e067a8f0ab93282a1e832820" Oct 31 19:41:28.349396 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.342694 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istiod-ca-cert\") pod \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " Oct 31 19:41:28.349396 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.342805 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-workload-certs\") pod \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " Oct 31 19:41:28.349396 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.342859 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/005cc55c-8e13-47d1-aa18-ddb1fb79432c-kube-api-access-dxjjs\" (UniqueName: \"kubernetes.io/projected/005cc55c-8e13-47d1-aa18-ddb1fb79432c-kube-api-access-dxjjs\") pod \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " Oct 31 19:41:28.349396 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.342903 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-credential-socket\") pod \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " Oct 31 19:41:28.349396 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.343022 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-podinfo\") pod \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " Oct 31 19:41:28.349396 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.343073 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-workload-socket\") pod \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " Oct 31 19:41:28.349396 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.343259 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-envoy\") pod \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " Oct 31 19:41:28.349396 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.343289 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-token\" (UniqueName: \"kubernetes.io/projected/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-token\") pod \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " Oct 31 19:41:28.349396 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.343432 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-data\") pod \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\" (UID: \"005cc55c-8e13-47d1-aa18-ddb1fb79432c\") " Oct 31 19:41:28.349396 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.348452 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-workload-certs" pod "005cc55c-8e13-47d1-aa18-ddb1fb79432c" (UID: "005cc55c-8e13-47d1-aa18-ddb1fb79432c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:28.350017 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.349812 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-credential-socket" pod "005cc55c-8e13-47d1-aa18-ddb1fb79432c" (UID: "005cc55c-8e13-47d1-aa18-ddb1fb79432c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:28.354498 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.353962 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-data" pod "005cc55c-8e13-47d1-aa18-ddb1fb79432c" (UID: "005cc55c-8e13-47d1-aa18-ddb1fb79432c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:28.354498 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.354230 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-workload-socket" pod "005cc55c-8e13-47d1-aa18-ddb1fb79432c" (UID: "005cc55c-8e13-47d1-aa18-ddb1fb79432c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:28.368631 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.368592 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/005cc55c-8e13-47d1-aa18-ddb1fb79432c-kube-api-access-dxjjs" pod "005cc55c-8e13-47d1-aa18-ddb1fb79432c" (UID: "005cc55c-8e13-47d1-aa18-ddb1fb79432c"). InnerVolumeSpecName "kube-api-access-dxjjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:28.374165 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.374110 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:28.383469 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.383419 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-podinfo" pod "005cc55c-8e13-47d1-aa18-ddb1fb79432c" (UID: "005cc55c-8e13-47d1-aa18-ddb1fb79432c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:41:28.386221 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.386026 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.425974 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.407851 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-token" pod "005cc55c-8e13-47d1-aa18-ddb1fb79432c" (UID: "005cc55c-8e13-47d1-aa18-ddb1fb79432c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:28.425974 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.407957 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-envoy" pod "005cc55c-8e13-47d1-aa18-ddb1fb79432c" (UID: "005cc55c-8e13-47d1-aa18-ddb1fb79432c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:28.629990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.488359 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.629990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.488392 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.629990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.488407 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.629990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.488422 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.629990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.488436 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxjjs\" (UniqueName: \"kubernetes.io/projected/005cc55c-8e13-47d1-aa18-ddb1fb79432c-kube-api-access-dxjjs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.629990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.488450 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.629990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.488464 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/005cc55c-8e13-47d1-aa18-ddb1fb79432c-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.629990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.488478 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/005cc55c-8e13-47d1-aa18-ddb1fb79432c-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.629990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.488530 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:28.629990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.588367 4910 scope.go:122] "RemoveContainer" containerID="c9491a09b25c87e60a2dacfc94e634611f8d7975f3f2e26489b25440cf4b66d6" Oct 31 19:41:28.629990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.620007 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:28.688583 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.688536 4910 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 31 19:41:28.688928 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.688813 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" podUID="00f5bd2c-c4f2-4854-8169-dd65ad5d301d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 31 19:41:28.691171 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.690510 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:41:28.698927 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.696118 4910 scope.go:122] "RemoveContainer" containerID="d40a2731d2e1ceb6bddcad5fbd1d49993fdc9c2aedf8bf5280fd43f6b67ed8ac" Oct 31 19:41:28.702926 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.702778 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9561493-df06-4f24-878e-f5d1255ac41b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9561493-df06-4f24-878e-f5d1255ac41b-istiod-ca-cert\") pod \"f9561493-df06-4f24-878e-f5d1255ac41b\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " Oct 31 19:41:28.703650 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.703105 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-workload-certs\") pod \"f9561493-df06-4f24-878e-f5d1255ac41b\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " Oct 31 19:41:28.703650 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.703176 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-istio-envoy\") pod \"f9561493-df06-4f24-878e-f5d1255ac41b\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " Oct 31 19:41:28.703650 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.703240 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9561493-df06-4f24-878e-f5d1255ac41b-kube-api-access-hs587\" (UniqueName: \"kubernetes.io/projected/f9561493-df06-4f24-878e-f5d1255ac41b-kube-api-access-hs587\") pod \"f9561493-df06-4f24-878e-f5d1255ac41b\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " Oct 31 19:41:28.703650 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.703275 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9561493-df06-4f24-878e-f5d1255ac41b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9561493-df06-4f24-878e-f5d1255ac41b-istio-podinfo\") pod \"f9561493-df06-4f24-878e-f5d1255ac41b\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " Oct 31 19:41:28.703650 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.703440 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-istio-data\") pod \"f9561493-df06-4f24-878e-f5d1255ac41b\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " Oct 31 19:41:28.703650 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.703488 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-workload-socket\") pod \"f9561493-df06-4f24-878e-f5d1255ac41b\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " Oct 31 19:41:28.703650 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.703579 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-credential-socket\") pod \"f9561493-df06-4f24-878e-f5d1255ac41b\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " Oct 31 19:41:28.704072 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.703694 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9561493-df06-4f24-878e-f5d1255ac41b-istio-token\" (UniqueName: \"kubernetes.io/projected/f9561493-df06-4f24-878e-f5d1255ac41b-istio-token\") pod \"f9561493-df06-4f24-878e-f5d1255ac41b\" (UID: \"f9561493-df06-4f24-878e-f5d1255ac41b\") " Oct 31 19:41:28.710220 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.710190 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-workload-certs" pod "f9561493-df06-4f24-878e-f5d1255ac41b" (UID: "f9561493-df06-4f24-878e-f5d1255ac41b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:28.710536 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.710514 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9561493-df06-4f24-878e-f5d1255ac41b-istiod-ca-cert" pod "f9561493-df06-4f24-878e-f5d1255ac41b" (UID: "f9561493-df06-4f24-878e-f5d1255ac41b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:41:28.711498 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.711297 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9561493-df06-4f24-878e-f5d1255ac41b-istio-token" pod "f9561493-df06-4f24-878e-f5d1255ac41b" (UID: "f9561493-df06-4f24-878e-f5d1255ac41b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:28.711498 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.711316 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-istio-data" pod "f9561493-df06-4f24-878e-f5d1255ac41b" (UID: "f9561493-df06-4f24-878e-f5d1255ac41b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:28.711722 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.711691 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-workload-socket" pod "f9561493-df06-4f24-878e-f5d1255ac41b" (UID: "f9561493-df06-4f24-878e-f5d1255ac41b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:28.711835 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.711797 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-credential-socket" pod "f9561493-df06-4f24-878e-f5d1255ac41b" (UID: "f9561493-df06-4f24-878e-f5d1255ac41b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:28.715558 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.715477 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:41:28.802288 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.780732 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9561493-df06-4f24-878e-f5d1255ac41b-istio-podinfo" pod "f9561493-df06-4f24-878e-f5d1255ac41b" (UID: "f9561493-df06-4f24-878e-f5d1255ac41b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:41:28.802288 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.786640 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9561493-df06-4f24-878e-f5d1255ac41b-kube-api-access-hs587" pod "f9561493-df06-4f24-878e-f5d1255ac41b" (UID: "f9561493-df06-4f24-878e-f5d1255ac41b"). InnerVolumeSpecName "kube-api-access-hs587". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:28.802288 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.786872 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-istio-envoy" pod "f9561493-df06-4f24-878e-f5d1255ac41b" (UID: "f9561493-df06-4f24-878e-f5d1255ac41b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:28.802288 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.801771 4910 scope.go:122] "RemoveContainer" containerID="9d1c823c6925f1ab683f3f5a436323f906f59f17d4115922a296ef3de285a0e0" Oct 31 19:41:28.807590 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.806191 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-credential-socket\") pod \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " Oct 31 19:41:28.807590 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.806334 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-workload-socket\") pod \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " Oct 31 19:41:28.807590 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.806483 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-token\" (UniqueName: \"kubernetes.io/projected/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-token\") pod \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " Oct 31 19:41:28.807590 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.806557 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-data\") pod \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " Oct 31 19:41:28.807590 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.806611 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istiod-ca-cert\") pod \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " Oct 31 19:41:28.807590 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.806659 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-workload-certs\") pod \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " Oct 31 19:41:28.807590 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.806695 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-envoy\") pod \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " Oct 31 19:41:28.807590 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.806775 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-kube-api-access-wqs64\" (UniqueName: \"kubernetes.io/projected/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-kube-api-access-wqs64\") pod \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " Oct 31 19:41:28.807590 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.806810 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-podinfo\") pod \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\" (UID: \"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16\") " Oct 31 19:41:28.809088 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.808563 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs587\" (UniqueName: \"kubernetes.io/projected/f9561493-df06-4f24-878e-f5d1255ac41b-kube-api-access-hs587\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.809088 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.808591 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9561493-df06-4f24-878e-f5d1255ac41b-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.809088 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.808606 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.809088 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.808621 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.809088 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.808636 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.809088 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.808649 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9561493-df06-4f24-878e-f5d1255ac41b-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.809088 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.808663 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9561493-df06-4f24-878e-f5d1255ac41b-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.809088 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.808678 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.809088 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.808691 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9561493-df06-4f24-878e-f5d1255ac41b-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.811776 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.811298 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-workload-certs" pod "6ab4bc46-8c70-4f5a-8e0d-f852190f8c16" (UID: "6ab4bc46-8c70-4f5a-8e0d-f852190f8c16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:28.811776 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.811594 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istiod-ca-cert" pod "6ab4bc46-8c70-4f5a-8e0d-f852190f8c16" (UID: "6ab4bc46-8c70-4f5a-8e0d-f852190f8c16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:41:28.812017 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.811790 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-credential-socket" pod "6ab4bc46-8c70-4f5a-8e0d-f852190f8c16" (UID: "6ab4bc46-8c70-4f5a-8e0d-f852190f8c16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:28.812155 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.812105 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-workload-socket" pod "6ab4bc46-8c70-4f5a-8e0d-f852190f8c16" (UID: "6ab4bc46-8c70-4f5a-8e0d-f852190f8c16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:28.812309 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.812287 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-podinfo" pod "6ab4bc46-8c70-4f5a-8e0d-f852190f8c16" (UID: "6ab4bc46-8c70-4f5a-8e0d-f852190f8c16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:41:28.812735 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.812711 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-data" pod "6ab4bc46-8c70-4f5a-8e0d-f852190f8c16" (UID: "6ab4bc46-8c70-4f5a-8e0d-f852190f8c16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:28.816590 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.816556 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-envoy" pod "6ab4bc46-8c70-4f5a-8e0d-f852190f8c16" (UID: "6ab4bc46-8c70-4f5a-8e0d-f852190f8c16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:28.817160 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.817080 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-token" pod "6ab4bc46-8c70-4f5a-8e0d-f852190f8c16" (UID: "6ab4bc46-8c70-4f5a-8e0d-f852190f8c16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:28.824633 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.824589 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-kube-api-access-wqs64" pod "6ab4bc46-8c70-4f5a-8e0d-f852190f8c16" (UID: "6ab4bc46-8c70-4f5a-8e0d-f852190f8c16"). InnerVolumeSpecName "kube-api-access-wqs64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:28.850539 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.849679 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:41:28.872870 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.872840 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf"] Oct 31 19:41:28.881628 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.881529 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-2zkcf"] Oct 31 19:41:28.902682 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.901695 4910 scope.go:122] "RemoveContainer" containerID="ad1a9a9e89e144c56863eac13e70e8b50eff089d96858192c0bc62c97d2bc165" Oct 31 19:41:28.911348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.909508 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-token\" (UniqueName: \"kubernetes.io/projected/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-token\") pod \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " Oct 31 19:41:28.911348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.909573 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-credential-socket\") pod \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " Oct 31 19:41:28.911348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.909607 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-envoy\") pod \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " Oct 31 19:41:28.911348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.909650 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-kube-api-access-8v7b7\" (UniqueName: \"kubernetes.io/projected/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-kube-api-access-8v7b7\") pod \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " Oct 31 19:41:28.911348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.909721 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istiod-ca-cert\") pod \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " Oct 31 19:41:28.911348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.909792 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-workload-certs\") pod \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " Oct 31 19:41:28.911348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.909862 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-workload-socket\") pod \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " Oct 31 19:41:28.911348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.909933 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-data\") pod \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " Oct 31 19:41:28.911348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.910117 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-podinfo\") pod \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\" (UID: \"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94\") " Oct 31 19:41:28.911348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.910891 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.911348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.910960 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.911348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.910979 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.911348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.910995 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.911348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.911009 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.911348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.911024 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqs64\" (UniqueName: \"kubernetes.io/projected/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-kube-api-access-wqs64\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.911348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.911038 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.911348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.911053 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.911348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.911066 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:28.913238 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.913210 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-workload-certs" pod "09020e9b-b4e8-4cbc-946e-73f3f7e8fc94" (UID: "09020e9b-b4e8-4cbc-946e-73f3f7e8fc94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:28.913458 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.913440 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-credential-socket" pod "09020e9b-b4e8-4cbc-946e-73f3f7e8fc94" (UID: "09020e9b-b4e8-4cbc-946e-73f3f7e8fc94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:28.914062 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.913656 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-workload-socket" pod "09020e9b-b4e8-4cbc-946e-73f3f7e8fc94" (UID: "09020e9b-b4e8-4cbc-946e-73f3f7e8fc94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:28.914062 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.913943 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istiod-ca-cert" pod "09020e9b-b4e8-4cbc-946e-73f3f7e8fc94" (UID: "09020e9b-b4e8-4cbc-946e-73f3f7e8fc94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:41:28.914062 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.914038 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-data" pod "09020e9b-b4e8-4cbc-946e-73f3f7e8fc94" (UID: "09020e9b-b4e8-4cbc-946e-73f3f7e8fc94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:28.915216 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.915187 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-kube-api-access-8v7b7" pod "09020e9b-b4e8-4cbc-946e-73f3f7e8fc94" (UID: "09020e9b-b4e8-4cbc-946e-73f3f7e8fc94"). InnerVolumeSpecName "kube-api-access-8v7b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:28.922222 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.922159 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-token" pod "09020e9b-b4e8-4cbc-946e-73f3f7e8fc94" (UID: "09020e9b-b4e8-4cbc-946e-73f3f7e8fc94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:28.922417 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.922385 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-podinfo" pod "09020e9b-b4e8-4cbc-946e-73f3f7e8fc94" (UID: "09020e9b-b4e8-4cbc-946e-73f3f7e8fc94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:41:28.957934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.948271 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-envoy" pod "09020e9b-b4e8-4cbc-946e-73f3f7e8fc94" (UID: "09020e9b-b4e8-4cbc-946e-73f3f7e8fc94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:28.965195 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.964264 4910 scope.go:122] "RemoveContainer" containerID="cc71dc7e40aab32e4c594451954bed8b145a7c986df063f9def589efb096bac5" Oct 31 19:41:28.983744 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.983718 4910 scope.go:122] "RemoveContainer" containerID="7c3495796b13f4ef5dc6c9200ae3f011863eef208c684cd3cebae59c114d4882" Oct 31 19:41:28.999815 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:28.999788 4910 scope.go:122] "RemoveContainer" containerID="a0ec8840783025046c13d0a7b093c07e4ff6fc0d2810bf63476f2685c21eaacf" Oct 31 19:41:29.000187 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:41:29.000159 4910 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"a0ec8840783025046c13d0a7b093c07e4ff6fc0d2810bf63476f2685c21eaacf\": rpc error: code = NotFound desc = could not find container \"a0ec8840783025046c13d0a7b093c07e4ff6fc0d2810bf63476f2685c21eaacf\": container with ID starting with a0ec8840783025046c13d0a7b093c07e4ff6fc0d2810bf63476f2685c21eaacf not found: ID does not exist" containerID="a0ec8840783025046c13d0a7b093c07e4ff6fc0d2810bf63476f2685c21eaacf" Oct 31 19:41:29.000300 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.000190 4910 scope.go:122] "RemoveContainer" containerID="8fd6ef98fd815b05f578dcd71cd6f3149b518c87f582ecabc658901d64b35528" Oct 31 19:41:29.012087 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.012062 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:29.012087 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.012087 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:29.012286 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.012099 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:29.012286 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.012109 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:29.012286 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.012117 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:29.012286 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.012125 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:29.012286 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.012133 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:29.012286 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.012141 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v7b7\" (UniqueName: \"kubernetes.io/projected/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-kube-api-access-8v7b7\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:29.012286 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.012150 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:29.014294 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.014274 4910 scope.go:122] "RemoveContainer" containerID="161d9b6e897960d783778eeda7411ac7003d3c50a3d9d9744881c72234ba4806" Oct 31 19:41:29.064216 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.064151 4910 scope.go:122] "RemoveContainer" containerID="938c793e482ab097f4b2404a129f15fdc2085114d23e5feed4c8864227b63a86" Oct 31 19:41:29.300743 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.300705 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" event={"ID":"09020e9b-b4e8-4cbc-946e-73f3f7e8fc94","Type":"ContainerDied","Data":"408581dc132d3db05ee34d971321e4cae2657021a9cb89063ccf6cdd6f740ac9"} Oct 31 19:41:29.301065 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.300719 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx" Oct 31 19:41:29.368668 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.368628 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" event={"ID":"6ab4bc46-8c70-4f5a-8e0d-f852190f8c16","Type":"ContainerDied","Data":"034e7e8358a5fa7276efb23ae1526420b255b00985bb2e3b25d16768d67bb097"} Oct 31 19:41:29.369492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.369467 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h" Oct 31 19:41:29.369778 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.369753 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" event={"ID":"f9561493-df06-4f24-878e-f5d1255ac41b","Type":"ContainerDied","Data":"3bda33a7e6f859544934a2016bcb87a2b66fb11f833cff07df2ef8d19554dcdd"} Oct 31 19:41:29.369989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.369900 4910 scope.go:122] "RemoveContainer" containerID="42d858a26dd2495bb3c2775ccee03fa7d591627c25db84bd9eef77f8831fb972" Oct 31 19:41:29.389830 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.389060 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr" Oct 31 19:41:29.395515 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:41:29.395482 4910 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@8-26632-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9561493_df06_4f24_878e_f5d1255ac41b.slice/crio-3bda33a7e6f859544934a2016bcb87a2b66fb11f833cff07df2ef8d19554dcdd\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507b2fb7_a22a_4cac_9d4a_23199f7fcbff.slice/crio-8ecb05670fd8cd8251e58cd0e3bdb1a8db6549d99b0a85c8015e2a152950365c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09020e9b_b4e8_4cbc_946e_73f3f7e8fc94.slice\": RecentStats: unable to find data in memory cache]" Oct 31 19:41:29.400077 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.399794 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx"] Oct 31 19:41:29.408003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.407980 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-kj2wx"] Oct 31 19:41:29.468274 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.468240 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h"] Oct 31 19:41:29.514402 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.514374 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-85w4h"] Oct 31 19:41:29.583074 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.583042 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr"] Oct 31 19:41:29.600095 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.599637 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-ffmnr"] Oct 31 19:41:29.686936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.686876 4910 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 31 19:41:29.687107 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.686979 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" podUID="00f5bd2c-c4f2-4854-8169-dd65ad5d301d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 31 19:41:29.765476 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.765393 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="005cc55c-8e13-47d1-aa18-ddb1fb79432c" path="/var/lib/kubelet/pods/005cc55c-8e13-47d1-aa18-ddb1fb79432c/volumes" Oct 31 19:41:29.766155 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.766136 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09020e9b-b4e8-4cbc-946e-73f3f7e8fc94" path="/var/lib/kubelet/pods/09020e9b-b4e8-4cbc-946e-73f3f7e8fc94/volumes" Oct 31 19:41:29.767042 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.767016 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ab4bc46-8c70-4f5a-8e0d-f852190f8c16" path="/var/lib/kubelet/pods/6ab4bc46-8c70-4f5a-8e0d-f852190f8c16/volumes" Oct 31 19:41:29.771618 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:29.771574 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9561493-df06-4f24-878e-f5d1255ac41b" path="/var/lib/kubelet/pods/f9561493-df06-4f24-878e-f5d1255ac41b/volumes" Oct 31 19:41:30.401844 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:30.401805 4910 generic.go:356] "Generic (PLEG): container finished" podID="507b2fb7-a22a-4cac-9d4a-23199f7fcbff" containerID="8ecb05670fd8cd8251e58cd0e3bdb1a8db6549d99b0a85c8015e2a152950365c" exitCode=0 Oct 31 19:41:30.402317 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:30.401880 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" event={"ID":"507b2fb7-a22a-4cac-9d4a-23199f7fcbff","Type":"ContainerDied","Data":"8ecb05670fd8cd8251e58cd0e3bdb1a8db6549d99b0a85c8015e2a152950365c"} Oct 31 19:41:30.402492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:30.402473 4910 scope.go:122] "RemoveContainer" containerID="8ecb05670fd8cd8251e58cd0e3bdb1a8db6549d99b0a85c8015e2a152950365c" Oct 31 19:41:30.404599 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:30.404565 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:30.404736 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:30.404652 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:30.404736 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:30.404705 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:30.687575 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:30.687547 4910 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 31 19:41:30.687708 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:30.687597 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" podUID="00f5bd2c-c4f2-4854-8169-dd65ad5d301d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 31 19:41:31.558935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:31.555612 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" event={"ID":"507b2fb7-a22a-4cac-9d4a-23199f7fcbff","Type":"ContainerStarted","Data":"0ee1fe21413c867adee167d9aac19f036cf7a8de9dc3f7c257ca2d48b0d929f5"} Oct 31 19:41:31.575249 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:31.575220 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:41:31.576665 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:31.576627 4910 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 19:41:31.577199 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:31.576869 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" podUID="507b2fb7-a22a-4cac-9d4a-23199f7fcbff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 19:41:31.687756 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:31.687683 4910 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 31 19:41:31.687756 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:31.687752 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" podUID="00f5bd2c-c4f2-4854-8169-dd65ad5d301d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 31 19:41:32.617936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:32.616776 4910 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 19:41:32.617936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:32.616840 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" podUID="507b2fb7-a22a-4cac-9d4a-23199f7fcbff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 19:41:32.736959 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:32.706067 4910 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 31 19:41:32.753147 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:32.753105 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" podUID="00f5bd2c-c4f2-4854-8169-dd65ad5d301d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 31 19:41:33.009983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.009476 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:41:33.149938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.149724 4910 generic.go:356] "Generic (PLEG): container finished" podID="6f712963-3098-4e5d-b62d-df829fd9a55b" containerID="98d6b358e9ba3ea26ccc7695e8377610d0e4a0468e232e9aafc77855b4cdebe6" exitCode=0 Oct 31 19:41:33.150156 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.150135 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" event={"ID":"6f712963-3098-4e5d-b62d-df829fd9a55b","Type":"ContainerDied","Data":"98d6b358e9ba3ea26ccc7695e8377610d0e4a0468e232e9aafc77855b4cdebe6"} Oct 31 19:41:33.215539 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.155398 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:41:33.215539 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.159128 4910 generic.go:356] "Generic (PLEG): container finished" podID="26ae8974-942b-4bd3-9ca9-f047209ab0fa" containerID="d5ca693c42416ee872e735614fbde751da9997d42c679a8f1207cce9f047313e" exitCode=0 Oct 31 19:41:33.215539 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.159404 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" event={"ID":"26ae8974-942b-4bd3-9ca9-f047209ab0fa","Type":"ContainerDied","Data":"d5ca693c42416ee872e735614fbde751da9997d42c679a8f1207cce9f047313e"} Oct 31 19:41:33.215539 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.159475 4910 scope.go:122] "RemoveContainer" containerID="e5e35193844c1dd27ba0c3731bd00c02d0979f62c443b1682e9e77ea3fe97e88" Oct 31 19:41:33.350418 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.350344 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8"] Oct 31 19:41:33.352291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.352237 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" podUID="5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8" containerName="istio-proxy" containerID="cri-o://25cbe54f224369bb6112062e2f930c6be73fab0040c3ebf55c5ca0e56a147dbe" gracePeriod=30 Oct 31 19:41:33.479933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.479870 4910 generic.go:356] "Generic (PLEG): container finished" podID="69edbb82-5f76-4f2d-b70a-d99a96ea5817" containerID="aff1762f981bda8ee76ed72e7e6766186f23f87cdce941d1fe7d573cc7951da6" exitCode=0 Oct 31 19:41:33.480338 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.480314 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" event={"ID":"69edbb82-5f76-4f2d-b70a-d99a96ea5817","Type":"ContainerDied","Data":"aff1762f981bda8ee76ed72e7e6766186f23f87cdce941d1fe7d573cc7951da6"} Oct 31 19:41:33.509591 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.508704 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh"] Oct 31 19:41:33.509591 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.508807 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg"] Oct 31 19:41:33.509591 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.509104 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" podUID="d7def544-b959-41db-a2b3-a66b68a81d10" containerName="istio-proxy" containerID="cri-o://5f4dccb6d6a00184dbb07aa13cefb8695cbe3993d1ef9ab2bde648081399a098" gracePeriod=30 Oct 31 19:41:33.557118 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.519619 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" podUID="465829b5-8a02-4035-b06d-a283ba96ae0f" containerName="istio-proxy" containerID="cri-o://3fc8bf7e4c936c02e8b32d083d2fbe8a5e90b1ca1a8810fd153183d2bb091b3e" gracePeriod=30 Oct 31 19:41:33.582707 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.562783 4910 generic.go:356] "Generic (PLEG): container finished" podID="df7e0301-710e-4fcb-8e00-6bc3ddccfedf" containerID="ef3b9adf524171a46ebda8465dbf13911e6ddb80ddfb0d6338e60f23060aa3f9" exitCode=0 Oct 31 19:41:33.582707 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.579822 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" Oct 31 19:41:33.582707 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.580086 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf" event={"ID":"df7e0301-710e-4fcb-8e00-6bc3ddccfedf","Type":"ContainerDied","Data":"ef3b9adf524171a46ebda8465dbf13911e6ddb80ddfb0d6338e60f23060aa3f9"} Oct 31 19:41:33.582707 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.581186 4910 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 19:41:33.582707 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.581236 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" podUID="507b2fb7-a22a-4cac-9d4a-23199f7fcbff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 19:41:33.584459 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.584427 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-token\" (UniqueName: \"kubernetes.io/projected/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-token\") pod \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " Oct 31 19:41:33.584670 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.584645 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-podinfo\") pod \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " Oct 31 19:41:33.584897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.584876 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-workload-certs\") pod \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " Oct 31 19:41:33.585115 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.585098 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-kube-api-access-wqccl\" (UniqueName: \"kubernetes.io/projected/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-kube-api-access-wqccl\") pod \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " Oct 31 19:41:33.585322 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.585307 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istiod-ca-cert\") pod \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " Oct 31 19:41:33.585456 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.585442 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-credential-socket\") pod \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " Oct 31 19:41:33.585664 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.585649 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-data\") pod \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " Oct 31 19:41:33.586042 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.586027 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-envoy\") pod \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " Oct 31 19:41:33.586236 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.586221 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-workload-socket\") pod \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\" (UID: \"df7e0301-710e-4fcb-8e00-6bc3ddccfedf\") " Oct 31 19:41:33.596176 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.596142 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-workload-socket" pod "df7e0301-710e-4fcb-8e00-6bc3ddccfedf" (UID: "df7e0301-710e-4fcb-8e00-6bc3ddccfedf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:33.599928 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.599890 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:33.601805 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.601774 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istiod-ca-cert" pod "df7e0301-710e-4fcb-8e00-6bc3ddccfedf" (UID: "df7e0301-710e-4fcb-8e00-6bc3ddccfedf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:41:33.602095 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.602069 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-workload-certs" pod "df7e0301-710e-4fcb-8e00-6bc3ddccfedf" (UID: "df7e0301-710e-4fcb-8e00-6bc3ddccfedf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:33.627438 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.627408 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-credential-socket" pod "df7e0301-710e-4fcb-8e00-6bc3ddccfedf" (UID: "df7e0301-710e-4fcb-8e00-6bc3ddccfedf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:33.671020 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.670981 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-data" pod "df7e0301-710e-4fcb-8e00-6bc3ddccfedf" (UID: "df7e0301-710e-4fcb-8e00-6bc3ddccfedf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:33.685344 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.684962 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-kube-api-access-wqccl" pod "df7e0301-710e-4fcb-8e00-6bc3ddccfedf" (UID: "df7e0301-710e-4fcb-8e00-6bc3ddccfedf"). InnerVolumeSpecName "kube-api-access-wqccl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:33.685344 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.685096 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-token" pod "df7e0301-710e-4fcb-8e00-6bc3ddccfedf" (UID: "df7e0301-710e-4fcb-8e00-6bc3ddccfedf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:33.687226 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.687206 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:41:33.691141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.691114 4910 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 31 19:41:33.691546 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.691516 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" podUID="00f5bd2c-c4f2-4854-8169-dd65ad5d301d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 31 19:41:33.708971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.701330 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-podinfo" pod "df7e0301-710e-4fcb-8e00-6bc3ddccfedf" (UID: "df7e0301-710e-4fcb-8e00-6bc3ddccfedf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:41:33.715760 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.712123 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:33.715760 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.712151 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:33.715760 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.712168 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:33.715760 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.712190 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqccl\" (UniqueName: \"kubernetes.io/projected/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-kube-api-access-wqccl\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:33.715760 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.712206 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:33.715760 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.712229 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:33.715760 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.712245 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:33.715760 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.713649 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:33.716701 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.716681 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:41:33.797288 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.797250 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-envoy" pod "df7e0301-710e-4fcb-8e00-6bc3ddccfedf" (UID: "df7e0301-710e-4fcb-8e00-6bc3ddccfedf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:33.813870 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.813746 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-workload-socket\") pod \"6f712963-3098-4e5d-b62d-df829fd9a55b\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " Oct 31 19:41:33.813870 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.813816 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-token\" (UniqueName: \"kubernetes.io/projected/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-token\") pod \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " Oct 31 19:41:33.814150 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.814071 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-data\") pod \"6f712963-3098-4e5d-b62d-df829fd9a55b\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " Oct 31 19:41:33.814220 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.814160 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-workload-socket\") pod \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " Oct 31 19:41:33.814279 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.814238 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-workload-certs\") pod \"6f712963-3098-4e5d-b62d-df829fd9a55b\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " Oct 31 19:41:33.814341 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.814273 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69edbb82-5f76-4f2d-b70a-d99a96ea5817-kube-api-access-84rt6\" (UniqueName: \"kubernetes.io/projected/69edbb82-5f76-4f2d-b70a-d99a96ea5817-kube-api-access-84rt6\") pod \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " Oct 31 19:41:33.814428 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.814410 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-podinfo\") pod \"6f712963-3098-4e5d-b62d-df829fd9a55b\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " Oct 31 19:41:33.814508 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.814493 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-envoy\") pod \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " Oct 31 19:41:33.814695 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.814644 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-workload-certs\") pod \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " Oct 31 19:41:33.814786 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.814728 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-credential-socket\") pod \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " Oct 31 19:41:33.814843 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.814800 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-credential-socket\") pod \"6f712963-3098-4e5d-b62d-df829fd9a55b\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " Oct 31 19:41:33.814894 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.814861 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f712963-3098-4e5d-b62d-df829fd9a55b-kube-api-access-5wslr\" (UniqueName: \"kubernetes.io/projected/6f712963-3098-4e5d-b62d-df829fd9a55b-kube-api-access-5wslr\") pod \"6f712963-3098-4e5d-b62d-df829fd9a55b\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " Oct 31 19:41:33.814983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.814949 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f712963-3098-4e5d-b62d-df829fd9a55b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f712963-3098-4e5d-b62d-df829fd9a55b-istiod-ca-cert\") pod \"6f712963-3098-4e5d-b62d-df829fd9a55b\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " Oct 31 19:41:33.815475 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.815128 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-podinfo\") pod \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " Oct 31 19:41:33.815475 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.815241 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-envoy\") pod \"6f712963-3098-4e5d-b62d-df829fd9a55b\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " Oct 31 19:41:33.815475 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.815277 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-token\" (UniqueName: \"kubernetes.io/projected/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-token\") pod \"6f712963-3098-4e5d-b62d-df829fd9a55b\" (UID: \"6f712963-3098-4e5d-b62d-df829fd9a55b\") " Oct 31 19:41:33.815475 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.815318 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-data\") pod \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " Oct 31 19:41:33.815475 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.815394 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istiod-ca-cert\") pod \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\" (UID: \"69edbb82-5f76-4f2d-b70a-d99a96ea5817\") " Oct 31 19:41:33.816231 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.816177 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-workload-certs" pod "69edbb82-5f76-4f2d-b70a-d99a96ea5817" (UID: "69edbb82-5f76-4f2d-b70a-d99a96ea5817"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:33.836233 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.836134 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-credential-socket" pod "69edbb82-5f76-4f2d-b70a-d99a96ea5817" (UID: "69edbb82-5f76-4f2d-b70a-d99a96ea5817"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:33.836644 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.836621 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-credential-socket" pod "6f712963-3098-4e5d-b62d-df829fd9a55b" (UID: "6f712963-3098-4e5d-b62d-df829fd9a55b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:33.873943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.864166 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-workload-socket" pod "69edbb82-5f76-4f2d-b70a-d99a96ea5817" (UID: "69edbb82-5f76-4f2d-b70a-d99a96ea5817"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:33.873943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.864389 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-workload-certs" pod "6f712963-3098-4e5d-b62d-df829fd9a55b" (UID: "6f712963-3098-4e5d-b62d-df829fd9a55b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:33.873943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.867116 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-workload-socket" pod "6f712963-3098-4e5d-b62d-df829fd9a55b" (UID: "6f712963-3098-4e5d-b62d-df829fd9a55b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:33.874660 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.874630 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-data" pod "6f712963-3098-4e5d-b62d-df829fd9a55b" (UID: "6f712963-3098-4e5d-b62d-df829fd9a55b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:33.916403 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.916366 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-data" pod "69edbb82-5f76-4f2d-b70a-d99a96ea5817" (UID: "69edbb82-5f76-4f2d-b70a-d99a96ea5817"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:33.917009 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.916982 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istiod-ca-cert" pod "69edbb82-5f76-4f2d-b70a-d99a96ea5817" (UID: "69edbb82-5f76-4f2d-b70a-d99a96ea5817"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:41:33.917875 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:33.917850 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f712963-3098-4e5d-b62d-df829fd9a55b-istiod-ca-cert" pod "6f712963-3098-4e5d-b62d-df829fd9a55b" (UID: "6f712963-3098-4e5d-b62d-df829fd9a55b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:41:34.060948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.049248 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.060948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.054111 4910 scope.go:122] "RemoveContainer" containerID="ef3b9adf524171a46ebda8465dbf13911e6ddb80ddfb0d6338e60f23060aa3f9" Oct 31 19:41:34.147125 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.099788 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.147125 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.099833 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.147125 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.099849 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.147125 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.099865 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.147125 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.099879 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.147125 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.099894 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.147125 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.099925 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df7e0301-710e-4fcb-8e00-6bc3ddccfedf-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.203608 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.203282 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-envoy\") pod \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " Oct 31 19:41:34.203608 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.203350 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-workload-certs\") pod \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " Oct 31 19:41:34.203608 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.203447 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-podinfo\") pod \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " Oct 31 19:41:34.203608 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.203503 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-data\") pod \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " Oct 31 19:41:34.203608 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.203550 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ae8974-942b-4bd3-9ca9-f047209ab0fa-kube-api-access-5bkqs\" (UniqueName: \"kubernetes.io/projected/26ae8974-942b-4bd3-9ca9-f047209ab0fa-kube-api-access-5bkqs\") pod \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " Oct 31 19:41:34.204170 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.203642 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istiod-ca-cert\") pod \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " Oct 31 19:41:34.204170 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.203793 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-credential-socket\") pod \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " Oct 31 19:41:34.204170 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.204034 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-token\" (UniqueName: \"kubernetes.io/projected/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-token\") pod \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " Oct 31 19:41:34.204406 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.204276 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-workload-socket\") pod \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\" (UID: \"26ae8974-942b-4bd3-9ca9-f047209ab0fa\") " Oct 31 19:41:34.205482 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.205446 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.429078 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.429047 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f712963-3098-4e5d-b62d-df829fd9a55b-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.445003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.444972 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.445609 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.209181 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-podinfo" pod "69edbb82-5f76-4f2d-b70a-d99a96ea5817" (UID: "69edbb82-5f76-4f2d-b70a-d99a96ea5817"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:41:34.450589 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.221272 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-workload-socket" pod "26ae8974-942b-4bd3-9ca9-f047209ab0fa" (UID: "26ae8974-942b-4bd3-9ca9-f047209ab0fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:34.452853 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.230749 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istiod-ca-cert" pod "26ae8974-942b-4bd3-9ca9-f047209ab0fa" (UID: "26ae8974-942b-4bd3-9ca9-f047209ab0fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:41:34.487968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.297282 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-credential-socket" pod "26ae8974-942b-4bd3-9ca9-f047209ab0fa" (UID: "26ae8974-942b-4bd3-9ca9-f047209ab0fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:34.487968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.298443 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-workload-certs" pod "26ae8974-942b-4bd3-9ca9-f047209ab0fa" (UID: "26ae8974-942b-4bd3-9ca9-f047209ab0fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:34.487968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.402097 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-data" pod "26ae8974-942b-4bd3-9ca9-f047209ab0fa" (UID: "26ae8974-942b-4bd3-9ca9-f047209ab0fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:34.487968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.438091 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69edbb82-5f76-4f2d-b70a-d99a96ea5817-kube-api-access-84rt6" pod "69edbb82-5f76-4f2d-b70a-d99a96ea5817" (UID: "69edbb82-5f76-4f2d-b70a-d99a96ea5817"). InnerVolumeSpecName "kube-api-access-84rt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:34.487968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.444066 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f712963-3098-4e5d-b62d-df829fd9a55b-kube-api-access-5wslr" pod "6f712963-3098-4e5d-b62d-df829fd9a55b" (UID: "6f712963-3098-4e5d-b62d-df829fd9a55b"). InnerVolumeSpecName "kube-api-access-5wslr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:34.487968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.432926 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h"] Oct 31 19:41:34.487968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.475206 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-token" pod "6f712963-3098-4e5d-b62d-df829fd9a55b" (UID: "6f712963-3098-4e5d-b62d-df829fd9a55b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:34.496948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.488871 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-token" pod "69edbb82-5f76-4f2d-b70a-d99a96ea5817" (UID: "69edbb82-5f76-4f2d-b70a-d99a96ea5817"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:34.496948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.488980 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-envoy" pod "6f712963-3098-4e5d-b62d-df829fd9a55b" (UID: "6f712963-3098-4e5d-b62d-df829fd9a55b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:34.496948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.489049 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-podinfo" pod "6f712963-3098-4e5d-b62d-df829fd9a55b" (UID: "6f712963-3098-4e5d-b62d-df829fd9a55b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:41:34.496948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.489113 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-envoy" pod "69edbb82-5f76-4f2d-b70a-d99a96ea5817" (UID: "69edbb82-5f76-4f2d-b70a-d99a96ea5817"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:34.510166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.498157 4910 cpu_manager.go:414] "containerMap: removing container" podUID="09020e9b-b4e8-4cbc-946e-73f3f7e8fc94" containerName="istio-proxy" Oct 31 19:41:34.510166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.498187 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09020e9b-b4e8-4cbc-946e-73f3f7e8fc94" containerName="istio-proxy" Oct 31 19:41:34.510166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.498417 4910 cpu_manager.go:414] "containerMap: removing container" podUID="26ae8974-942b-4bd3-9ca9-f047209ab0fa" containerName="istio-proxy" Oct 31 19:41:34.510166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.498433 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ae8974-942b-4bd3-9ca9-f047209ab0fa" containerName="istio-proxy" Oct 31 19:41:34.510166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.498623 4910 cpu_manager.go:414] "containerMap: removing container" podUID="df7e0301-710e-4fcb-8e00-6bc3ddccfedf" containerName="istio-proxy" Oct 31 19:41:34.510166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.498641 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df7e0301-710e-4fcb-8e00-6bc3ddccfedf" containerName="istio-proxy" Oct 31 19:41:34.510166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.498859 4910 cpu_manager.go:414] "containerMap: removing container" podUID="f9561493-df06-4f24-878e-f5d1255ac41b" containerName="istio-proxy" Oct 31 19:41:34.510166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.498873 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9561493-df06-4f24-878e-f5d1255ac41b" containerName="istio-proxy" Oct 31 19:41:34.510166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.499094 4910 cpu_manager.go:414] "containerMap: removing container" podUID="6ab4bc46-8c70-4f5a-8e0d-f852190f8c16" containerName="istio-proxy" Oct 31 19:41:34.510166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.499105 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab4bc46-8c70-4f5a-8e0d-f852190f8c16" containerName="istio-proxy" Oct 31 19:41:34.510166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.499122 4910 cpu_manager.go:414] "containerMap: removing container" podUID="6f712963-3098-4e5d-b62d-df829fd9a55b" containerName="istio-proxy" Oct 31 19:41:34.510166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.499132 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f712963-3098-4e5d-b62d-df829fd9a55b" containerName="istio-proxy" Oct 31 19:41:34.510166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.499146 4910 cpu_manager.go:414] "containerMap: removing container" podUID="005cc55c-8e13-47d1-aa18-ddb1fb79432c" containerName="istio-proxy" Oct 31 19:41:34.510166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.499154 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="005cc55c-8e13-47d1-aa18-ddb1fb79432c" containerName="istio-proxy" Oct 31 19:41:34.535316 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.515147 4910 cpu_manager.go:414] "containerMap: removing container" podUID="69edbb82-5f76-4f2d-b70a-d99a96ea5817" containerName="istio-proxy" Oct 31 19:41:34.535316 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.515346 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69edbb82-5f76-4f2d-b70a-d99a96ea5817" containerName="istio-proxy" Oct 31 19:41:34.535316 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.516742 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ae8974-942b-4bd3-9ca9-f047209ab0fa" containerName="istio-proxy" Oct 31 19:41:34.535316 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.516779 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="09020e9b-b4e8-4cbc-946e-73f3f7e8fc94" containerName="istio-proxy" Oct 31 19:41:34.535316 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.516793 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="26ae8974-942b-4bd3-9ca9-f047209ab0fa" containerName="istio-proxy" Oct 31 19:41:34.535316 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.517017 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="df7e0301-710e-4fcb-8e00-6bc3ddccfedf" containerName="istio-proxy" Oct 31 19:41:34.535316 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.517035 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9561493-df06-4f24-878e-f5d1255ac41b" containerName="istio-proxy" Oct 31 19:41:34.535316 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.517055 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab4bc46-8c70-4f5a-8e0d-f852190f8c16" containerName="istio-proxy" Oct 31 19:41:34.535316 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.517232 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f712963-3098-4e5d-b62d-df829fd9a55b" containerName="istio-proxy" Oct 31 19:41:34.535316 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.517245 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="005cc55c-8e13-47d1-aa18-ddb1fb79432c" containerName="istio-proxy" Oct 31 19:41:34.535316 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.517257 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="69edbb82-5f76-4f2d-b70a-d99a96ea5817" containerName="istio-proxy" Oct 31 19:41:34.535316 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.517591 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-envoy" pod "26ae8974-942b-4bd3-9ca9-f047209ab0fa" (UID: "26ae8974-942b-4bd3-9ca9-f047209ab0fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:34.535316 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.517822 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-podinfo" pod "26ae8974-942b-4bd3-9ca9-f047209ab0fa" (UID: "26ae8974-942b-4bd3-9ca9-f047209ab0fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:41:34.550492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.550219 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wslr\" (UniqueName: \"kubernetes.io/projected/6f712963-3098-4e5d-b62d-df829fd9a55b-kube-api-access-5wslr\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.550492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.550258 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.550492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.550275 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.550492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.550291 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.550492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.550305 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.550492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.550320 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.550492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.550336 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.550492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.550349 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.550492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.550363 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.550492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.550377 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.550492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.550392 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84rt6\" (UniqueName: \"kubernetes.io/projected/69edbb82-5f76-4f2d-b70a-d99a96ea5817-kube-api-access-84rt6\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.550492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.550409 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.550492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.550423 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f712963-3098-4e5d-b62d-df829fd9a55b-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.550492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.550437 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69edbb82-5f76-4f2d-b70a-d99a96ea5817-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.550492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.550451 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26ae8974-942b-4bd3-9ca9-f047209ab0fa-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.591934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.589985 4910 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 19:41:34.591934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.590047 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" podUID="507b2fb7-a22a-4cac-9d4a-23199f7fcbff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 19:41:34.603788 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.603719 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-token" pod "26ae8974-942b-4bd3-9ca9-f047209ab0fa" (UID: "26ae8974-942b-4bd3-9ca9-f047209ab0fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:34.634082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.616943 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26ae8974-942b-4bd3-9ca9-f047209ab0fa-kube-api-access-5bkqs" pod "26ae8974-942b-4bd3-9ca9-f047209ab0fa" (UID: "26ae8974-942b-4bd3-9ca9-f047209ab0fa"). InnerVolumeSpecName "kube-api-access-5bkqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:34.634082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.622663 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" Oct 31 19:41:34.634082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.630016 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h"] Oct 31 19:41:34.634082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.630065 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf"] Oct 31 19:41:34.634082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.630086 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-tqljf"] Oct 31 19:41:34.634082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.630110 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272" event={"ID":"6f712963-3098-4e5d-b62d-df829fd9a55b","Type":"ContainerDied","Data":"30af2674b6781fc938b70edb42eb864d5e9fb869b2ff5b97013611353c6a75a3"} Oct 31 19:41:34.634082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.630154 4910 scope.go:122] "RemoveContainer" containerID="98d6b358e9ba3ea26ccc7695e8377610d0e4a0468e232e9aafc77855b4cdebe6" Oct 31 19:41:34.634082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.630543 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:34.655210 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.638507 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" event={"ID":"26ae8974-942b-4bd3-9ca9-f047209ab0fa","Type":"ContainerDied","Data":"ddbf29cbfca49d8f83560ff58cb07b5d99ebe0a3477a5769ce033cceb6c443b1"} Oct 31 19:41:34.655210 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.638615 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw" Oct 31 19:41:34.655210 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.641694 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" event={"ID":"69edbb82-5f76-4f2d-b70a-d99a96ea5817","Type":"ContainerDied","Data":"9cba9c3396bf67099758e5b2eb143034adb8fbe27e77c802a8ef3d26a33b2c2d"} Oct 31 19:41:34.655210 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.641758 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l" Oct 31 19:41:34.656952 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.656643 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26ae8974-942b-4bd3-9ca9-f047209ab0fa-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.656952 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.656684 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bkqs\" (UniqueName: \"kubernetes.io/projected/26ae8974-942b-4bd3-9ca9-f047209ab0fa-kube-api-access-5bkqs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:34.712005 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.711977 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a4e312-openshift-default-dockercfg-kdghj\"" Oct 31 19:41:34.764945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.758370 4910 scope.go:122] "RemoveContainer" containerID="d5ca693c42416ee872e735614fbde751da9997d42c679a8f1207cce9f047313e" Oct 31 19:41:34.780978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.780880 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l"] Oct 31 19:41:34.794946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.794160 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-hmf8l"] Oct 31 19:41:34.796820 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.796788 4910 scope.go:122] "RemoveContainer" containerID="aff1762f981bda8ee76ed72e7e6766186f23f87cdce941d1fe7d573cc7951da6" Oct 31 19:41:34.805136 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.805088 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272"] Oct 31 19:41:34.855261 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.855188 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-sw272"] Oct 31 19:41:34.872043 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.865326 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-istio-envoy\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:34.872043 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.865412 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-istio-data\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:34.872043 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.865460 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-workload-socket\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:34.872043 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.865543 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8d77e7-9866-404f-83fc-1809898cec90-istio-podinfo\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:34.872043 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.865612 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8d77e7-9866-404f-83fc-1809898cec90-istiod-ca-cert\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:34.872043 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.865672 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-workload-certs\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:34.872043 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.865741 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8d77e7-9866-404f-83fc-1809898cec90-istio-token\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:34.872043 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.865793 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-credential-socket\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:34.872043 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.865878 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77wml\" (UniqueName: \"kubernetes.io/projected/5e8d77e7-9866-404f-83fc-1809898cec90-kube-api-access-77wml\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:34.888894 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.888836 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw"] Oct 31 19:41:34.933376 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.930345 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-w8vmw"] Oct 31 19:41:34.971082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.970984 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8d77e7-9866-404f-83fc-1809898cec90-istio-podinfo\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:34.971881 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.971856 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8d77e7-9866-404f-83fc-1809898cec90-istiod-ca-cert\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:34.983933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.972754 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8d77e7-9866-404f-83fc-1809898cec90-istiod-ca-cert\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:34.984551 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.984524 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-workload-certs\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:34.985145 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.985038 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-workload-certs\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:34.985360 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.985338 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8d77e7-9866-404f-83fc-1809898cec90-istio-token\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:34.985852 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:34.985829 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-credential-socket\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:35.001306 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.001113 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-credential-socket\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:35.001549 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.001527 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77wml\" (UniqueName: \"kubernetes.io/projected/5e8d77e7-9866-404f-83fc-1809898cec90-kube-api-access-77wml\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:35.002286 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.002252 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-istio-envoy\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:35.006293 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.005417 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-istio-envoy\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:35.006293 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.006262 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-istio-data\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:35.006467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.006320 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-workload-socket\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:35.006843 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.006752 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-workload-socket\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:35.006843 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.006814 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-istio-data\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:35.010850 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.010824 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8d77e7-9866-404f-83fc-1809898cec90-istio-podinfo\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:35.011514 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.011468 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8d77e7-9866-404f-83fc-1809898cec90-istio-token\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:35.015329 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.015305 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e73256-5lm7h"] Oct 31 19:41:35.017385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.017362 4910 cpu_manager.go:414] "containerMap: removing container" podUID="26ae8974-942b-4bd3-9ca9-f047209ab0fa" containerName="istio-proxy" Oct 31 19:41:35.017521 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.017504 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26ae8974-942b-4bd3-9ca9-f047209ab0fa" containerName="istio-proxy" Oct 31 19:41:35.043934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.036195 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77wml\" (UniqueName: \"kubernetes.io/projected/5e8d77e7-9866-404f-83fc-1809898cec90-kube-api-access-77wml\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:35.045565 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.045535 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e73256-5lm7h" Oct 31 19:41:35.055490 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.054768 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:41:35.055490 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.054954 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:41:35.087875 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.087818 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e73256-5lm7h"] Oct 31 19:41:35.088442 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.088376 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:35.092322 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.092300 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:35.094085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.093598 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:35.094293 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.094231 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:41:35.107801 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.107714 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a480bcfb-f234-4e55-a2fb-15990c28a0a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e73256-5lm7h\" (UID: \"a480bcfb-f234-4e55-a2fb-15990c28a0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e73256-5lm7h" Oct 31 19:41:35.108145 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.108121 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a480bcfb-f234-4e55-a2fb-15990c28a0a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e73256-5lm7h\" (UID: \"a480bcfb-f234-4e55-a2fb-15990c28a0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e73256-5lm7h" Oct 31 19:41:35.108373 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.108355 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxxqt\" (UniqueName: \"kubernetes.io/projected/a480bcfb-f234-4e55-a2fb-15990c28a0a1-kube-api-access-cxxqt\") pod \"maas-api-revoke-keys-e2e-api-a-e73256-5lm7h\" (UID: \"a480bcfb-f234-4e55-a2fb-15990c28a0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e73256-5lm7h" Oct 31 19:41:35.211956 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.211353 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a480bcfb-f234-4e55-a2fb-15990c28a0a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e73256-5lm7h\" (UID: \"a480bcfb-f234-4e55-a2fb-15990c28a0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e73256-5lm7h" Oct 31 19:41:35.211956 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.211507 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a480bcfb-f234-4e55-a2fb-15990c28a0a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e73256-5lm7h\" (UID: \"a480bcfb-f234-4e55-a2fb-15990c28a0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e73256-5lm7h" Oct 31 19:41:35.211956 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.211586 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxxqt\" (UniqueName: \"kubernetes.io/projected/a480bcfb-f234-4e55-a2fb-15990c28a0a1-kube-api-access-cxxqt\") pod \"maas-api-revoke-keys-e2e-api-a-e73256-5lm7h\" (UID: \"a480bcfb-f234-4e55-a2fb-15990c28a0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e73256-5lm7h" Oct 31 19:41:35.212718 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.212688 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a480bcfb-f234-4e55-a2fb-15990c28a0a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e73256-5lm7h\" (UID: \"a480bcfb-f234-4e55-a2fb-15990c28a0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e73256-5lm7h" Oct 31 19:41:35.216367 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.216338 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a480bcfb-f234-4e55-a2fb-15990c28a0a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e73256-5lm7h\" (UID: \"a480bcfb-f234-4e55-a2fb-15990c28a0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e73256-5lm7h" Oct 31 19:41:35.289318 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.286789 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxxqt\" (UniqueName: \"kubernetes.io/projected/a480bcfb-f234-4e55-a2fb-15990c28a0a1-kube-api-access-cxxqt\") pod \"maas-api-revoke-keys-e2e-api-a-e73256-5lm7h\" (UID: \"a480bcfb-f234-4e55-a2fb-15990c28a0a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e73256-5lm7h" Oct 31 19:41:35.479277 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.479244 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e73256-5lm7h" Oct 31 19:41:35.499044 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.499011 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h"] Oct 31 19:41:35.499431 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:41:35.499379 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8d77e7_9866_404f_83fc_1809898cec90.slice/crio-ac5511460855bd3e01ada49cb531b58760d955e85174d46329d967e0a90c0591 WatchSource:0}: Error finding container ac5511460855bd3e01ada49cb531b58760d955e85174d46329d967e0a90c0591: Status 404 returned error can't find the container with id ac5511460855bd3e01ada49cb531b58760d955e85174d46329d967e0a90c0591 Oct 31 19:41:35.503061 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.503019 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:35.503156 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.503112 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:35.503214 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.503151 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:35.566561 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.566523 4910 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 19:41:35.566737 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.566583 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" podUID="507b2fb7-a22a-4cac-9d4a-23199f7fcbff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 19:41:35.709469 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.709424 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" event={"ID":"5e8d77e7-9866-404f-83fc-1809898cec90","Type":"ContainerStarted","Data":"ac5511460855bd3e01ada49cb531b58760d955e85174d46329d967e0a90c0591"} Oct 31 19:41:35.791129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.791095 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26ae8974-942b-4bd3-9ca9-f047209ab0fa" path="/var/lib/kubelet/pods/26ae8974-942b-4bd3-9ca9-f047209ab0fa/volumes" Oct 31 19:41:35.792119 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.792096 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69edbb82-5f76-4f2d-b70a-d99a96ea5817" path="/var/lib/kubelet/pods/69edbb82-5f76-4f2d-b70a-d99a96ea5817/volumes" Oct 31 19:41:35.793175 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.793154 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f712963-3098-4e5d-b62d-df829fd9a55b" path="/var/lib/kubelet/pods/6f712963-3098-4e5d-b62d-df829fd9a55b/volumes" Oct 31 19:41:35.794605 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.794584 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df7e0301-710e-4fcb-8e00-6bc3ddccfedf" path="/var/lib/kubelet/pods/df7e0301-710e-4fcb-8e00-6bc3ddccfedf/volumes" Oct 31 19:41:35.912876 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:35.908158 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e73256-5lm7h"] Oct 31 19:41:36.570944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:36.569216 4910 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 19:41:36.570944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:36.569277 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" podUID="507b2fb7-a22a-4cac-9d4a-23199f7fcbff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 19:41:36.687489 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:36.686190 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:41:36.767864 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:36.767822 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e73256-5lm7h" event={"ID":"a480bcfb-f234-4e55-a2fb-15990c28a0a1","Type":"ContainerStarted","Data":"f6b5f827976801d42108f1d68eacbf99da94673783e085b8ce71ccb546edae38"} Oct 31 19:41:36.768442 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:36.767889 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e73256-5lm7h" event={"ID":"a480bcfb-f234-4e55-a2fb-15990c28a0a1","Type":"ContainerStarted","Data":"e60b110756f3d34df75a8d8ff5fc7b8a40522e927df78244014839424d45fbfb"} Oct 31 19:41:36.770423 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:36.770390 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" event={"ID":"5e8d77e7-9866-404f-83fc-1809898cec90","Type":"ContainerStarted","Data":"887e5ff8622f885dc948a0d730a1d9c5e85176dbf723a89aaf4433bfd83853b6"} Oct 31 19:41:36.796235 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:36.796166 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e73256-5lm7h" podStartSLOduration=2.796151567 podStartE2EDuration="2.796151567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:36.792161595 +0000 UTC m=+1331.154981991" watchObservedRunningTime="2026-10-31 19:41:36.796151567 +0000 UTC m=+1331.158971938" Oct 31 19:41:36.837723 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:36.837602 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" podStartSLOduration=3.837562213 podStartE2EDuration="3.837562213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:36.823180365 +0000 UTC m=+1331.186000741" watchObservedRunningTime="2026-10-31 19:41:36.837562213 +0000 UTC m=+1331.200382586" Oct 31 19:41:37.095111 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:37.095022 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:37.100835 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:37.100801 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b66ef818-fwll4"] Oct 31 19:41:37.103186 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:37.102845 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:37.105621 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:37.105597 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b66ef818-fwll4"] Oct 31 19:41:37.566070 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:37.566039 4910 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 19:41:37.566231 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:37.566090 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" podUID="507b2fb7-a22a-4cac-9d4a-23199f7fcbff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 19:41:37.762379 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:37.762342 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ad9721c-495b-413e-b60d-29e34471c03d" path="/var/lib/kubelet/pods/5ad9721c-495b-413e-b60d-29e34471c03d/volumes" Oct 31 19:41:37.783449 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:37.783414 4910 generic.go:356] "Generic (PLEG): container finished" podID="a480bcfb-f234-4e55-a2fb-15990c28a0a1" containerID="f6b5f827976801d42108f1d68eacbf99da94673783e085b8ce71ccb546edae38" exitCode=0 Oct 31 19:41:37.783925 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:37.783500 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e73256-5lm7h" event={"ID":"a480bcfb-f234-4e55-a2fb-15990c28a0a1","Type":"ContainerDied","Data":"f6b5f827976801d42108f1d68eacbf99da94673783e085b8ce71ccb546edae38"} Oct 31 19:41:37.783996 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:37.783957 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:37.785131 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:37.785111 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:41:38.496280 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.496244 4910 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:41:38.496451 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.496311 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" podUID="d7def544-b959-41db-a2b3-a66b68a81d10" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:41:38.572955 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.572560 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:41:38.574415 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.574389 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:41:38.774053 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.773857 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:38.838509 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.838468 4910 generic.go:356] "Generic (PLEG): container finished" podID="465829b5-8a02-4035-b06d-a283ba96ae0f" containerID="3fc8bf7e4c936c02e8b32d083d2fbe8a5e90b1ca1a8810fd153183d2bb091b3e" exitCode=0 Oct 31 19:41:38.839015 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.838564 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" event={"ID":"465829b5-8a02-4035-b06d-a283ba96ae0f","Type":"ContainerDied","Data":"3fc8bf7e4c936c02e8b32d083d2fbe8a5e90b1ca1a8810fd153183d2bb091b3e"} Oct 31 19:41:38.907082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.905300 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-token\" (UniqueName: \"kubernetes.io/projected/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-token\") pod \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " Oct 31 19:41:38.907082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.905374 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-workload-socket\") pod \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " Oct 31 19:41:38.907082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.905505 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-envoy\") pod \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " Oct 31 19:41:38.907082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.905573 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-workload-certs\") pod \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " Oct 31 19:41:38.907082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.905623 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istiod-ca-cert\") pod \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " Oct 31 19:41:38.907082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.905783 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-kube-api-access-42lmz\" (UniqueName: \"kubernetes.io/projected/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-kube-api-access-42lmz\") pod \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " Oct 31 19:41:38.907082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.905838 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-podinfo\") pod \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " Oct 31 19:41:38.907082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.905887 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-data\") pod \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " Oct 31 19:41:38.907082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.905958 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-credential-socket\") pod \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\" (UID: \"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8\") " Oct 31 19:41:38.908288 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.908018 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-credential-socket" pod "5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8" (UID: "5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:38.913223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.913195 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istiod-ca-cert" pod "5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8" (UID: "5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:41:38.913512 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.913433 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-token" pod "5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8" (UID: "5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:38.913641 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.913598 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-workload-certs" pod "5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8" (UID: "5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:38.914287 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.914266 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-data" pod "5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8" (UID: "5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:38.914433 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.914389 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-workload-socket" pod "5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8" (UID: "5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:38.914689 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.914662 4910 generic.go:356] "Generic (PLEG): container finished" podID="5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8" containerID="25cbe54f224369bb6112062e2f930c6be73fab0040c3ebf55c5ca0e56a147dbe" exitCode=0 Oct 31 19:41:38.914949 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.914778 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" event={"ID":"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8","Type":"ContainerDied","Data":"25cbe54f224369bb6112062e2f930c6be73fab0040c3ebf55c5ca0e56a147dbe"} Oct 31 19:41:38.914949 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.914829 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" event={"ID":"5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8","Type":"ContainerDied","Data":"0a2b202af9b0a2e79694445c882936fcb9e71c2861ca8f23730834fbbd213160"} Oct 31 19:41:38.914949 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.914858 4910 scope.go:122] "RemoveContainer" containerID="25cbe54f224369bb6112062e2f930c6be73fab0040c3ebf55c5ca0e56a147dbe" Oct 31 19:41:38.915176 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.915158 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8" Oct 31 19:41:38.916391 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.916163 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-envoy" pod "5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8" (UID: "5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:38.921974 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.920396 4910 generic.go:356] "Generic (PLEG): container finished" podID="d7def544-b959-41db-a2b3-a66b68a81d10" containerID="5f4dccb6d6a00184dbb07aa13cefb8695cbe3993d1ef9ab2bde648081399a098" exitCode=0 Oct 31 19:41:38.921974 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.921627 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" event={"ID":"d7def544-b959-41db-a2b3-a66b68a81d10","Type":"ContainerDied","Data":"5f4dccb6d6a00184dbb07aa13cefb8695cbe3993d1ef9ab2bde648081399a098"} Oct 31 19:41:38.927585 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.927514 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-kube-api-access-42lmz" pod "5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8" (UID: "5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8"). InnerVolumeSpecName "kube-api-access-42lmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:38.928046 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.928019 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-podinfo" pod "5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8" (UID: "5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:41:38.961825 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.961443 4910 scope.go:122] "RemoveContainer" containerID="25cbe54f224369bb6112062e2f930c6be73fab0040c3ebf55c5ca0e56a147dbe" Oct 31 19:41:38.962536 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:41:38.962457 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25cbe54f224369bb6112062e2f930c6be73fab0040c3ebf55c5ca0e56a147dbe\": container with ID starting with 25cbe54f224369bb6112062e2f930c6be73fab0040c3ebf55c5ca0e56a147dbe not found: ID does not exist" containerID="25cbe54f224369bb6112062e2f930c6be73fab0040c3ebf55c5ca0e56a147dbe" Oct 31 19:41:38.962536 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.962501 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25cbe54f224369bb6112062e2f930c6be73fab0040c3ebf55c5ca0e56a147dbe"} err="failed to get container status \"25cbe54f224369bb6112062e2f930c6be73fab0040c3ebf55c5ca0e56a147dbe\": rpc error: code = NotFound desc = could not find container \"25cbe54f224369bb6112062e2f930c6be73fab0040c3ebf55c5ca0e56a147dbe\": container with ID starting with 25cbe54f224369bb6112062e2f930c6be73fab0040c3ebf55c5ca0e56a147dbe not found: ID does not exist" Oct 31 19:41:38.981092 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:38.981059 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:39.013986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.011236 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.013986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.012088 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42lmz\" (UniqueName: \"kubernetes.io/projected/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-kube-api-access-42lmz\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.013986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.012107 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.013986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.012118 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.013986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.012127 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.013986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.012136 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.013986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.012146 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.013986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.012155 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.013986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.012163 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.066965 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.066691 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:41:39.113374 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.113341 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-podinfo\") pod \"465829b5-8a02-4035-b06d-a283ba96ae0f\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " Oct 31 19:41:39.113649 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.113630 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-token\" (UniqueName: \"kubernetes.io/projected/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-token\") pod \"465829b5-8a02-4035-b06d-a283ba96ae0f\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " Oct 31 19:41:39.114221 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.114197 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-workload-certs\") pod \"465829b5-8a02-4035-b06d-a283ba96ae0f\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " Oct 31 19:41:39.114333 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.114302 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/465829b5-8a02-4035-b06d-a283ba96ae0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/465829b5-8a02-4035-b06d-a283ba96ae0f-istiod-ca-cert\") pod \"465829b5-8a02-4035-b06d-a283ba96ae0f\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " Oct 31 19:41:39.114516 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.114498 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-data\") pod \"465829b5-8a02-4035-b06d-a283ba96ae0f\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " Oct 31 19:41:39.114608 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.114509 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-workload-certs" pod "465829b5-8a02-4035-b06d-a283ba96ae0f" (UID: "465829b5-8a02-4035-b06d-a283ba96ae0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:39.114608 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.114560 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-envoy\") pod \"465829b5-8a02-4035-b06d-a283ba96ae0f\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " Oct 31 19:41:39.114608 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.114599 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-credential-socket\") pod \"465829b5-8a02-4035-b06d-a283ba96ae0f\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " Oct 31 19:41:39.114828 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.114634 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/465829b5-8a02-4035-b06d-a283ba96ae0f-kube-api-access-ts7s7\" (UniqueName: \"kubernetes.io/projected/465829b5-8a02-4035-b06d-a283ba96ae0f-kube-api-access-ts7s7\") pod \"465829b5-8a02-4035-b06d-a283ba96ae0f\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " Oct 31 19:41:39.114828 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.114755 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-workload-socket\") pod \"465829b5-8a02-4035-b06d-a283ba96ae0f\" (UID: \"465829b5-8a02-4035-b06d-a283ba96ae0f\") " Oct 31 19:41:39.115262 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.115200 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-credential-socket" pod "465829b5-8a02-4035-b06d-a283ba96ae0f" (UID: "465829b5-8a02-4035-b06d-a283ba96ae0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:39.115605 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.115479 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-workload-socket" pod "465829b5-8a02-4035-b06d-a283ba96ae0f" (UID: "465829b5-8a02-4035-b06d-a283ba96ae0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:39.115760 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.115685 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-data" pod "465829b5-8a02-4035-b06d-a283ba96ae0f" (UID: "465829b5-8a02-4035-b06d-a283ba96ae0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:39.116135 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.116039 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/465829b5-8a02-4035-b06d-a283ba96ae0f-istiod-ca-cert" pod "465829b5-8a02-4035-b06d-a283ba96ae0f" (UID: "465829b5-8a02-4035-b06d-a283ba96ae0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:41:39.116135 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.116070 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.116135 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.116091 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.116135 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.116106 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.116135 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.116120 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.116647 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.116615 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-token" pod "465829b5-8a02-4035-b06d-a283ba96ae0f" (UID: "465829b5-8a02-4035-b06d-a283ba96ae0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:39.118353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.118320 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-envoy" pod "465829b5-8a02-4035-b06d-a283ba96ae0f" (UID: "465829b5-8a02-4035-b06d-a283ba96ae0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:39.118470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.118433 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465829b5-8a02-4035-b06d-a283ba96ae0f-kube-api-access-ts7s7" pod "465829b5-8a02-4035-b06d-a283ba96ae0f" (UID: "465829b5-8a02-4035-b06d-a283ba96ae0f"). InnerVolumeSpecName "kube-api-access-ts7s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:39.119233 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.119208 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-podinfo" pod "465829b5-8a02-4035-b06d-a283ba96ae0f" (UID: "465829b5-8a02-4035-b06d-a283ba96ae0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:41:39.151640 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.151615 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e73256-5lm7h" Oct 31 19:41:39.217122 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.217080 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7def544-b959-41db-a2b3-a66b68a81d10-kube-api-access-bn9fh\" (UniqueName: \"kubernetes.io/projected/d7def544-b959-41db-a2b3-a66b68a81d10-kube-api-access-bn9fh\") pod \"d7def544-b959-41db-a2b3-a66b68a81d10\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " Oct 31 19:41:39.217397 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.217374 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7def544-b959-41db-a2b3-a66b68a81d10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7def544-b959-41db-a2b3-a66b68a81d10-istiod-ca-cert\") pod \"d7def544-b959-41db-a2b3-a66b68a81d10\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " Oct 31 19:41:39.219368 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.219236 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7def544-b959-41db-a2b3-a66b68a81d10-istiod-ca-cert" pod "d7def544-b959-41db-a2b3-a66b68a81d10" (UID: "d7def544-b959-41db-a2b3-a66b68a81d10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:41:39.219658 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.219641 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-istio-envoy\") pod \"d7def544-b959-41db-a2b3-a66b68a81d10\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " Oct 31 19:41:39.220124 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.220106 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7def544-b959-41db-a2b3-a66b68a81d10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7def544-b959-41db-a2b3-a66b68a81d10-istio-podinfo\") pod \"d7def544-b959-41db-a2b3-a66b68a81d10\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " Oct 31 19:41:39.220580 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.220562 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-credential-socket\") pod \"d7def544-b959-41db-a2b3-a66b68a81d10\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " Oct 31 19:41:39.221124 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.221099 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7def544-b959-41db-a2b3-a66b68a81d10-istio-token\" (UniqueName: \"kubernetes.io/projected/d7def544-b959-41db-a2b3-a66b68a81d10-istio-token\") pod \"d7def544-b959-41db-a2b3-a66b68a81d10\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " Oct 31 19:41:39.221556 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.221530 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-workload-certs\") pod \"d7def544-b959-41db-a2b3-a66b68a81d10\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " Oct 31 19:41:39.221959 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.220986 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-credential-socket" pod "d7def544-b959-41db-a2b3-a66b68a81d10" (UID: "d7def544-b959-41db-a2b3-a66b68a81d10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:39.221959 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.221858 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-workload-certs" pod "d7def544-b959-41db-a2b3-a66b68a81d10" (UID: "d7def544-b959-41db-a2b3-a66b68a81d10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:39.222278 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.222257 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-istio-data\") pod \"d7def544-b959-41db-a2b3-a66b68a81d10\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " Oct 31 19:41:39.222670 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.222548 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-istio-data" pod "d7def544-b959-41db-a2b3-a66b68a81d10" (UID: "d7def544-b959-41db-a2b3-a66b68a81d10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:39.222873 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.222844 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7def544-b959-41db-a2b3-a66b68a81d10-istio-podinfo" pod "d7def544-b959-41db-a2b3-a66b68a81d10" (UID: "d7def544-b959-41db-a2b3-a66b68a81d10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:41:39.223192 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.223094 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-workload-socket\") pod \"d7def544-b959-41db-a2b3-a66b68a81d10\" (UID: \"d7def544-b959-41db-a2b3-a66b68a81d10\") " Oct 31 19:41:39.223615 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.223502 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-workload-socket" pod "d7def544-b959-41db-a2b3-a66b68a81d10" (UID: "d7def544-b959-41db-a2b3-a66b68a81d10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:39.227012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.226984 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.227123 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.227109 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts7s7\" (UniqueName: \"kubernetes.io/projected/465829b5-8a02-4035-b06d-a283ba96ae0f-kube-api-access-ts7s7\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.227182 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.227175 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7def544-b959-41db-a2b3-a66b68a81d10-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.227232 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.227225 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.227288 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.227281 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/465829b5-8a02-4035-b06d-a283ba96ae0f-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.227349 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.227342 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7def544-b959-41db-a2b3-a66b68a81d10-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.227402 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.227395 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.227449 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.227441 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.227496 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.227490 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/465829b5-8a02-4035-b06d-a283ba96ae0f-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.227552 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.227542 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.227877 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.227860 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.228432 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.228411 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7def544-b959-41db-a2b3-a66b68a81d10-istio-token" pod "d7def544-b959-41db-a2b3-a66b68a81d10" (UID: "d7def544-b959-41db-a2b3-a66b68a81d10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:39.229416 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.229382 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7def544-b959-41db-a2b3-a66b68a81d10-kube-api-access-bn9fh" pod "d7def544-b959-41db-a2b3-a66b68a81d10" (UID: "d7def544-b959-41db-a2b3-a66b68a81d10"). InnerVolumeSpecName "kube-api-access-bn9fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:39.229673 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.229657 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-istio-envoy" pod "d7def544-b959-41db-a2b3-a66b68a81d10" (UID: "d7def544-b959-41db-a2b3-a66b68a81d10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:41:39.261404 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.261371 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8"] Oct 31 19:41:39.265649 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.265619 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ht5g8"] Oct 31 19:41:39.328408 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.328371 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a480bcfb-f234-4e55-a2fb-15990c28a0a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a480bcfb-f234-4e55-a2fb-15990c28a0a1-maas-api-xks-ca\") pod \"a480bcfb-f234-4e55-a2fb-15990c28a0a1\" (UID: \"a480bcfb-f234-4e55-a2fb-15990c28a0a1\") " Oct 31 19:41:39.328594 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.328493 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a480bcfb-f234-4e55-a2fb-15990c28a0a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a480bcfb-f234-4e55-a2fb-15990c28a0a1-maas-api-openshift-ca\") pod \"a480bcfb-f234-4e55-a2fb-15990c28a0a1\" (UID: \"a480bcfb-f234-4e55-a2fb-15990c28a0a1\") " Oct 31 19:41:39.328741 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.328725 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a480bcfb-f234-4e55-a2fb-15990c28a0a1-kube-api-access-cxxqt\" (UniqueName: \"kubernetes.io/projected/a480bcfb-f234-4e55-a2fb-15990c28a0a1-kube-api-access-cxxqt\") pod \"a480bcfb-f234-4e55-a2fb-15990c28a0a1\" (UID: \"a480bcfb-f234-4e55-a2fb-15990c28a0a1\") " Oct 31 19:41:39.328825 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.328802 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a480bcfb-f234-4e55-a2fb-15990c28a0a1-maas-api-openshift-ca" pod "a480bcfb-f234-4e55-a2fb-15990c28a0a1" (UID: "a480bcfb-f234-4e55-a2fb-15990c28a0a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:41:39.329254 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.329232 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn9fh\" (UniqueName: \"kubernetes.io/projected/d7def544-b959-41db-a2b3-a66b68a81d10-kube-api-access-bn9fh\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.329254 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.329254 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7def544-b959-41db-a2b3-a66b68a81d10-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.329412 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.329264 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7def544-b959-41db-a2b3-a66b68a81d10-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.329412 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.329273 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a480bcfb-f234-4e55-a2fb-15990c28a0a1-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.330608 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.330585 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a480bcfb-f234-4e55-a2fb-15990c28a0a1-maas-api-xks-ca" pod "a480bcfb-f234-4e55-a2fb-15990c28a0a1" (UID: "a480bcfb-f234-4e55-a2fb-15990c28a0a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:41:39.330713 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.330687 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a480bcfb-f234-4e55-a2fb-15990c28a0a1-kube-api-access-cxxqt" pod "a480bcfb-f234-4e55-a2fb-15990c28a0a1" (UID: "a480bcfb-f234-4e55-a2fb-15990c28a0a1"). InnerVolumeSpecName "kube-api-access-cxxqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:41:39.431093 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.431059 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxxqt\" (UniqueName: \"kubernetes.io/projected/a480bcfb-f234-4e55-a2fb-15990c28a0a1-kube-api-access-cxxqt\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.431093 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.431090 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a480bcfb-f234-4e55-a2fb-15990c28a0a1-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:41:39.762742 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.762644 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8" path="/var/lib/kubelet/pods/5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8/volumes" Oct 31 19:41:39.932870 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.932837 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e73256-5lm7h" Oct 31 19:41:39.932870 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.932861 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e73256-5lm7h" event={"ID":"a480bcfb-f234-4e55-a2fb-15990c28a0a1","Type":"ContainerDied","Data":"e60b110756f3d34df75a8d8ff5fc7b8a40522e927df78244014839424d45fbfb"} Oct 31 19:41:39.933388 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.932961 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e60b110756f3d34df75a8d8ff5fc7b8a40522e927df78244014839424d45fbfb" Oct 31 19:41:39.934803 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.934652 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" event={"ID":"465829b5-8a02-4035-b06d-a283ba96ae0f","Type":"ContainerDied","Data":"6d41020e2c555694df431ea819edd34efc34b02d4f18d7ef530dc5aef10f0dee"} Oct 31 19:41:39.934803 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.934702 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh" Oct 31 19:41:39.934803 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.934711 4910 scope.go:122] "RemoveContainer" containerID="3fc8bf7e4c936c02e8b32d083d2fbe8a5e90b1ca1a8810fd153183d2bb091b3e" Oct 31 19:41:39.939452 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.939432 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" Oct 31 19:41:39.939452 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.939440 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg" event={"ID":"d7def544-b959-41db-a2b3-a66b68a81d10","Type":"ContainerDied","Data":"52029cd5dd830f32208a6165d80c956804b6dfd9f6160f1a9cce8939bd7ed095"} Oct 31 19:41:39.950842 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.950790 4910 scope.go:122] "RemoveContainer" containerID="5f4dccb6d6a00184dbb07aa13cefb8695cbe3993d1ef9ab2bde648081399a098" Oct 31 19:41:39.970536 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.970509 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh"] Oct 31 19:41:39.974814 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.974780 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-lrglh"] Oct 31 19:41:39.997993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:39.997965 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg"] Oct 31 19:41:40.002133 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:40.002107 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-5rfdg"] Oct 31 19:41:40.595670 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:40.595634 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j"] Oct 31 19:41:40.597241 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:40.597141 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d7def544-b959-41db-a2b3-a66b68a81d10" containerName="istio-proxy" Oct 31 19:41:40.597241 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:40.597187 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7def544-b959-41db-a2b3-a66b68a81d10" containerName="istio-proxy" Oct 31 19:41:40.597241 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:40.597231 4910 cpu_manager.go:414] "containerMap: removing container" podUID="a480bcfb-f234-4e55-a2fb-15990c28a0a1" containerName="revoke-keys" Oct 31 19:41:40.597241 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:40.597240 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a480bcfb-f234-4e55-a2fb-15990c28a0a1" containerName="revoke-keys" Oct 31 19:41:40.597544 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:40.597279 4910 cpu_manager.go:414] "containerMap: removing container" podUID="5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8" containerName="istio-proxy" Oct 31 19:41:40.597544 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:40.597287 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8" containerName="istio-proxy" Oct 31 19:41:40.597544 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:40.597330 4910 cpu_manager.go:414] "containerMap: removing container" podUID="465829b5-8a02-4035-b06d-a283ba96ae0f" containerName="istio-proxy" Oct 31 19:41:40.597544 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:40.597337 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="465829b5-8a02-4035-b06d-a283ba96ae0f" containerName="istio-proxy" Oct 31 19:41:40.598409 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:40.598382 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7def544-b959-41db-a2b3-a66b68a81d10" containerName="istio-proxy" Oct 31 19:41:40.598548 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:40.598443 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="a480bcfb-f234-4e55-a2fb-15990c28a0a1" containerName="revoke-keys" Oct 31 19:41:40.598548 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:40.598466 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="5152b948-1f8a-4ee8-aa7a-e7d2a6dc58e8" containerName="istio-proxy" Oct 31 19:41:40.598548 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:40.598479 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="465829b5-8a02-4035-b06d-a283ba96ae0f" containerName="istio-proxy" Oct 31 19:41:40.606813 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:40.606029 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j" Oct 31 19:41:40.620688 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:40.620656 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j"] Oct 31 19:41:40.649394 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:40.649365 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hb4g\" (UniqueName: \"kubernetes.io/projected/09ee3f2a-332e-4c98-9b21-c76f3903b69f-kube-api-access-9hb4g\") pod \"payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j\" (UID: \"09ee3f2a-332e-4c98-9b21-c76f3903b69f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j" Oct 31 19:41:40.649672 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:40.649648 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09ee3f2a-332e-4c98-9b21-c76f3903b69f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j\" (UID: \"09ee3f2a-332e-4c98-9b21-c76f3903b69f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j" Oct 31 19:41:40.699930 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:40.698640 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a4e312\"" Oct 31 19:41:40.721148 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:40.714160 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a4e312-dockercfg-g6knc\"" Oct 31 19:41:40.784986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:40.770268 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hb4g\" (UniqueName: \"kubernetes.io/projected/09ee3f2a-332e-4c98-9b21-c76f3903b69f-kube-api-access-9hb4g\") pod \"payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j\" (UID: \"09ee3f2a-332e-4c98-9b21-c76f3903b69f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j" Oct 31 19:41:40.784986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:40.770370 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09ee3f2a-332e-4c98-9b21-c76f3903b69f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j\" (UID: \"09ee3f2a-332e-4c98-9b21-c76f3903b69f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j" Oct 31 19:41:40.784986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:40.771420 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09ee3f2a-332e-4c98-9b21-c76f3903b69f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j\" (UID: \"09ee3f2a-332e-4c98-9b21-c76f3903b69f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j" Oct 31 19:41:40.830940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:40.822329 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hb4g\" (UniqueName: \"kubernetes.io/projected/09ee3f2a-332e-4c98-9b21-c76f3903b69f-kube-api-access-9hb4g\") pod \"payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j\" (UID: \"09ee3f2a-332e-4c98-9b21-c76f3903b69f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j" Oct 31 19:41:40.943980 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:40.943071 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j" Oct 31 19:41:41.297841 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:41:41.297809 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ee3f2a_332e_4c98_9b21_c76f3903b69f.slice/crio-7eef78e5d7e075acd5c2514123103b7c0008409819e59ab6886362a27ff34cdc WatchSource:0}: Error finding container 7eef78e5d7e075acd5c2514123103b7c0008409819e59ab6886362a27ff34cdc: Status 404 returned error can't find the container with id 7eef78e5d7e075acd5c2514123103b7c0008409819e59ab6886362a27ff34cdc Oct 31 19:41:41.298906 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:41.298882 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j"] Oct 31 19:41:41.757725 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:41.757684 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="465829b5-8a02-4035-b06d-a283ba96ae0f" path="/var/lib/kubelet/pods/465829b5-8a02-4035-b06d-a283ba96ae0f/volumes" Oct 31 19:41:41.758542 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:41.758521 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7def544-b959-41db-a2b3-a66b68a81d10" path="/var/lib/kubelet/pods/d7def544-b959-41db-a2b3-a66b68a81d10/volumes" Oct 31 19:41:42.026983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:42.026852 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j" event={"ID":"09ee3f2a-332e-4c98-9b21-c76f3903b69f","Type":"ContainerStarted","Data":"cd2371917831ea70df2741d58567ef8a93339d01d1b41ae865a4415ed9c429d3"} Oct 31 19:41:42.026983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:42.026957 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j" event={"ID":"09ee3f2a-332e-4c98-9b21-c76f3903b69f","Type":"ContainerStarted","Data":"7eef78e5d7e075acd5c2514123103b7c0008409819e59ab6886362a27ff34cdc"} Oct 31 19:41:42.027465 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:42.027162 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j" Oct 31 19:41:42.047737 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:42.047677 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j" podStartSLOduration=2.047660453 podStartE2EDuration="2.047660453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:42.04540768 +0000 UTC m=+1336.408228054" watchObservedRunningTime="2026-10-31 19:41:42.047660453 +0000 UTC m=+1336.410480825" Oct 31 19:41:42.737478 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:41:42.737430 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:41:44.784970 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.784934 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv"] Oct 31 19:41:44.789215 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.789192 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.809705 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.809679 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv"] Oct 31 19:41:44.869944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.869599 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istiod-ca-cert\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.869944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.869853 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-workload-certs\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.870376 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.870031 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-podinfo\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.887291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.871086 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbjbx\" (UniqueName: \"kubernetes.io/projected/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-kube-api-access-mbjbx\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.887291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.871316 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-workload-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.887291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.871385 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-envoy\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.887291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.871413 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-credential-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.887291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.871450 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-token\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.887291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.871592 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-data\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.973505 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.973461 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-data\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.973696 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.973576 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istiod-ca-cert\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.973696 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.973653 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-workload-certs\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.973840 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.973724 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-podinfo\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.973840 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.973779 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbjbx\" (UniqueName: \"kubernetes.io/projected/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-kube-api-access-mbjbx\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.974005 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.973927 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-workload-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.974213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.974020 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-envoy\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.974213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.974022 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-data\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.974213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.974057 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-credential-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.974213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.974092 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-token\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.974463 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.974295 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-workload-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.974463 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.974403 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istiod-ca-cert\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.974735 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.974714 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-workload-certs\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.975051 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.975017 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-credential-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.976682 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.976653 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-envoy\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.976798 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.976778 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-podinfo\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.986329 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.986308 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-token\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:44.986509 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:44.986492 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbjbx\" (UniqueName: \"kubernetes.io/projected/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-kube-api-access-mbjbx\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:45.106197 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:45.106157 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:45.417834 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:45.417739 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv"] Oct 31 19:41:45.479626 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:41:45.478300 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2cd527e_0cf0_4513_97b6_7f1f9d8d1584.slice/crio-65de04335cec6bc3a1e96da597fc4f1c7126abdbee1737e13420379ae3136b87 WatchSource:0}: Error finding container 65de04335cec6bc3a1e96da597fc4f1c7126abdbee1737e13420379ae3136b87: Status 404 returned error can't find the container with id 65de04335cec6bc3a1e96da597fc4f1c7126abdbee1737e13420379ae3136b87 Oct 31 19:41:45.502953 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:45.483243 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:45.502953 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:45.483348 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:45.502953 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:45.483392 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:46.066113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:46.066073 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" event={"ID":"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584","Type":"ContainerStarted","Data":"b390e60d26d488396a2d95fc5d1f493fd90143d1a0fe8aa9dc7680bb0d189b5f"} Oct 31 19:41:46.066667 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:46.066626 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" event={"ID":"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584","Type":"ContainerStarted","Data":"65de04335cec6bc3a1e96da597fc4f1c7126abdbee1737e13420379ae3136b87"} Oct 31 19:41:46.123932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:46.123277 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:46.138933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:46.138322 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" podStartSLOduration=2.138288803 podStartE2EDuration="2.138288803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:46.094275988 +0000 UTC m=+1340.457096374" watchObservedRunningTime="2026-10-31 19:41:46.138288803 +0000 UTC m=+1340.501109178" Oct 31 19:41:46.139265 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:46.139236 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 31 19:41:46.139427 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:46.139403 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" podUID="d2cd527e-0cf0-4513-97b6-7f1f9d8d1584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 31 19:41:46.883935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:46.882637 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj"] Oct 31 19:41:46.888292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:46.888262 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:46.893574 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:46.893547 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-278c04-openshift-default-dockercfg-rsckc\"" Oct 31 19:41:46.897693 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:46.895479 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj"] Oct 31 19:41:46.949975 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:46.949494 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-workload-certs\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:46.949975 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:46.949574 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-workload-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:46.949975 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:46.949690 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-envoy\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:46.949975 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:46.949726 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm5hm\" (UniqueName: \"kubernetes.io/projected/499b41f2-15df-4636-abbc-35fa6f8a164a-kube-api-access-dm5hm\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:46.949975 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:46.949815 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-token\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:46.949975 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:46.949854 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-data\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:46.949975 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:46.949929 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-podinfo\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:46.950516 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:46.950011 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/499b41f2-15df-4636-abbc-35fa6f8a164a-istiod-ca-cert\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:46.950516 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:46.950300 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-credential-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:47.117015 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:47.116971 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-workload-certs\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:47.136654 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:47.136572 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-workload-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:47.158938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:47.137000 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-envoy\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:47.159286 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:47.159254 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm5hm\" (UniqueName: \"kubernetes.io/projected/499b41f2-15df-4636-abbc-35fa6f8a164a-kube-api-access-dm5hm\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:47.159573 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:47.159532 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-token\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:47.159764 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:47.159746 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-data\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:47.160031 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:47.160005 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-podinfo\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:47.194206 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:47.194174 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/499b41f2-15df-4636-abbc-35fa6f8a164a-istiod-ca-cert\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:47.194610 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:47.194581 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-credential-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:47.194982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:47.178420 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-workload-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:47.195296 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:47.193063 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 31 19:41:47.195488 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:47.195446 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" podUID="d2cd527e-0cf0-4513-97b6-7f1f9d8d1584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 31 19:41:47.218937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:47.195753 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/499b41f2-15df-4636-abbc-35fa6f8a164a-istiod-ca-cert\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:47.290071 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:47.136447 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-workload-certs\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:47.290268 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:47.219517 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-credential-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:47.290396 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:47.178213 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-data\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:47.294394 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:47.294351 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-podinfo\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:47.317814 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:47.314009 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-envoy\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:47.318845 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:47.318791 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-token\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:47.321927 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:47.321861 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm5hm\" (UniqueName: \"kubernetes.io/projected/499b41f2-15df-4636-abbc-35fa6f8a164a-kube-api-access-dm5hm\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:47.530987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:47.522250 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:47.913575 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:41:47.913546 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod499b41f2_15df_4636_abbc_35fa6f8a164a.slice/crio-ce164ba30ac0c97c9d65abb7ef12f9bfeb5a9e1b318e8b2ab604b51cd61937c7 WatchSource:0}: Error finding container ce164ba30ac0c97c9d65abb7ef12f9bfeb5a9e1b318e8b2ab604b51cd61937c7: Status 404 returned error can't find the container with id ce164ba30ac0c97c9d65abb7ef12f9bfeb5a9e1b318e8b2ab604b51cd61937c7 Oct 31 19:41:47.914357 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:47.914333 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj"] Oct 31 19:41:47.916525 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:47.916488 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:47.916627 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:47.916572 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:47.916689 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:47.916640 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:41:48.109733 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:48.108777 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 31 19:41:48.109733 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:48.108834 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" podUID="d2cd527e-0cf0-4513-97b6-7f1f9d8d1584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 31 19:41:48.295503 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:48.295406 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" event={"ID":"499b41f2-15df-4636-abbc-35fa6f8a164a","Type":"ContainerStarted","Data":"ee5138499d94a28ba9e5fff1962310587ee29c357947bdfd6e3be726302de64c"} Oct 31 19:41:48.295967 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:48.295764 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" event={"ID":"499b41f2-15df-4636-abbc-35fa6f8a164a","Type":"ContainerStarted","Data":"ce164ba30ac0c97c9d65abb7ef12f9bfeb5a9e1b318e8b2ab604b51cd61937c7"} Oct 31 19:41:48.344983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:48.344905 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" podStartSLOduration=2.344885024 podStartE2EDuration="2.344885024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:48.334112849 +0000 UTC m=+1342.696933222" watchObservedRunningTime="2026-10-31 19:41:48.344885024 +0000 UTC m=+1342.707705400" Oct 31 19:41:48.538076 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:48.538036 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:49.107003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:49.106958 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 31 19:41:49.107003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:49.107028 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" podUID="d2cd527e-0cf0-4513-97b6-7f1f9d8d1584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 31 19:41:49.557117 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:49.557083 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:50.106752 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:50.106720 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 31 19:41:50.106907 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:50.106773 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" podUID="d2cd527e-0cf0-4513-97b6-7f1f9d8d1584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 31 19:41:50.332356 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:50.332318 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:50.338952 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:50.338902 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:41:51.107140 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:51.107108 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 31 19:41:51.107540 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:51.107161 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" podUID="d2cd527e-0cf0-4513-97b6-7f1f9d8d1584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 31 19:41:52.106825 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:52.106733 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 31 19:41:52.106825 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:52.106800 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" podUID="d2cd527e-0cf0-4513-97b6-7f1f9d8d1584" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 31 19:41:53.087946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:53.086141 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j" Oct 31 19:41:53.417840 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:53.417694 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:53.418486 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:53.418428 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:53.419763 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:53.419723 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 19:41:59.973052 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:41:59.972999 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:00.070735 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:00.070685 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:00.083098 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:00.083050 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:00.599607 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:00.599170 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:42:00.727294 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:00.727249 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:00.735025 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:00.733993 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:00.904006 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:00.903041 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:00.980936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:00.977704 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:00.980936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:00.978187 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:42:01.183963 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:01.170828 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:42:01.183963 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:01.171234 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:42:01.285036 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:01.284988 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:01.360064 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:01.360010 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:01.644967 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:01.619865 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:01.776055 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:01.776005 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:01.798936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:01.776887 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:42:02.056169 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:02.029611 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:02.060929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:02.030610 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:42:02.089936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:02.052736 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:02.089936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:02.052785 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:02.183983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:02.183937 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:02.343935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:02.336520 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded" Oct 31 19:42:02.370935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:02.360567 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:02.370935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:02.362214 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:42:02.597171 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:02.597067 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:42:02.598601 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:02.598565 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 19:42:02.620059 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:02.620012 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:42:02.640781 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:02.620477 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:02.642347 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:02.641088 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:42:02.665101 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:02.665037 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:02.747063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:02.747013 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:02.780954 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:02.780889 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:42:02.957994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:02.953977 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:02.993931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:02.990161 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:42:03.061937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:03.061228 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 19:42:03.061937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:03.061779 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:42:03.270527 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:03.266005 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:03.333932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:03.333101 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:03.452947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:03.445383 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:03.597706 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:03.597650 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:03.598442 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:03.598415 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:42:03.789705 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:03.600592 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:03.790389 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:03.624272 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:03.811975 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:03.704896 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:03.811975 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:03.749426 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:03.811975 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:03.773468 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:42:04.066936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.066548 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:04.157936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.151423 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:42:04.157936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.151606 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:42:04.157936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.151666 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:42:04.157936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.151726 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:42:04.157936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.151778 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:42:04.157936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.151825 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:42:04.191936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.184565 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:42:04.278716 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.278425 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:04.279299 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.279264 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:42:04.392496 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.392408 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p"] Oct 31 19:42:04.399127 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.399069 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:04.405744 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.405717 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:42:04.406852 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.406831 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.499934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.494498 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded" Oct 31 19:42:04.550935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.529096 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:42:04.603071 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.603030 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:04.608648 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.608544 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p"] Oct 31 19:42:04.615016 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.611595 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:04.624942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.616782 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:42:04.624942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.619371 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:42:04.643639 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.643548 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:42:04.669212 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.669180 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:42:04.680295 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.680256 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df617b70-0871-4423-b738-50476b66c833-istiod-ca-cert\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.680487 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.680469 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-workload-certs\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.680643 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.680563 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-workload-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.680643 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.680611 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzrbm\" (UniqueName: \"kubernetes.io/projected/df617b70-0871-4423-b738-50476b66c833-kube-api-access-bzrbm\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.680791 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.680663 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-istio-data\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.680791 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.680772 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-istio-envoy\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.680903 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.680839 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df617b70-0871-4423-b738-50476b66c833-istio-podinfo\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.680903 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.680875 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df617b70-0871-4423-b738-50476b66c833-istio-token\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.681047 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.681006 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-credential-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.697008 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.696980 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:42:04.712122 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.712091 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:42:04.716275 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.716251 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:42:04.783752 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.783711 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df617b70-0871-4423-b738-50476b66c833-istio-podinfo\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.783971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.783768 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df617b70-0871-4423-b738-50476b66c833-istio-token\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.783971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.783856 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-credential-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.784325 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.784299 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df617b70-0871-4423-b738-50476b66c833-istiod-ca-cert\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.784521 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.784498 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-workload-certs\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.784590 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.784572 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-workload-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.784654 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.784610 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzrbm\" (UniqueName: \"kubernetes.io/projected/df617b70-0871-4423-b738-50476b66c833-kube-api-access-bzrbm\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.784710 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.784667 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-istio-data\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.784815 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.784773 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-istio-envoy\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.786799 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.784900 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-workload-certs\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.786964 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.786838 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-credential-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.787424 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.787400 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-istio-data\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.787502 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.787445 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-workload-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.787502 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.787465 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df617b70-0871-4423-b738-50476b66c833-istiod-ca-cert\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.787726 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.787708 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df617b70-0871-4423-b738-50476b66c833-istio-podinfo\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.790080 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.789642 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-istio-envoy\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.802973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.802902 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df617b70-0871-4423-b738-50476b66c833-istio-token\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:04.803103 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:04.803068 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzrbm\" (UniqueName: \"kubernetes.io/projected/df617b70-0871-4423-b738-50476b66c833-kube-api-access-bzrbm\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:05.006903 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:05.006807 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:05.466859 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:05.466608 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p"] Oct 31 19:42:05.470050 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:42:05.470018 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf617b70_0871_4423_b738_50476b66c833.slice/crio-e65be1b7d0ff661aabd0e582be4eb6d4099aaaed5cd3a75d132dc83e51bf1a9a WatchSource:0}: Error finding container e65be1b7d0ff661aabd0e582be4eb6d4099aaaed5cd3a75d132dc83e51bf1a9a: Status 404 returned error can't find the container with id e65be1b7d0ff661aabd0e582be4eb6d4099aaaed5cd3a75d132dc83e51bf1a9a Oct 31 19:42:05.473526 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:05.473482 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:42:05.473607 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:05.473577 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:42:05.473652 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:05.473629 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:42:05.831927 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:05.817122 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" event={"ID":"df617b70-0871-4423-b738-50476b66c833","Type":"ContainerStarted","Data":"508db727dd9f175dee66c2f1aa91cf0cbb554a09f6947c83a70ef71c9bca48e5"} Oct 31 19:42:05.831927 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:05.817189 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e86eb251-mbhb4"] Oct 31 19:42:05.873075 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:05.872867 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e86eb251-mbhb4"] Oct 31 19:42:05.873075 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:05.873005 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86eb251-mbhb4" Oct 31 19:42:05.873075 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:05.873026 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" event={"ID":"df617b70-0871-4423-b738-50476b66c833","Type":"ContainerStarted","Data":"e65be1b7d0ff661aabd0e582be4eb6d4099aaaed5cd3a75d132dc83e51bf1a9a"} Oct 31 19:42:05.878145 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:05.878122 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:42:05.878425 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:05.878389 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:42:05.891492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:05.891449 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" podStartSLOduration=1.891436597 podStartE2EDuration="1.891436597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:05.885261182 +0000 UTC m=+1360.248081554" watchObservedRunningTime="2026-10-31 19:42:05.891436597 +0000 UTC m=+1360.254256968" Oct 31 19:42:05.912467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:05.912438 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29ded46-17c9-4189-8401-99218bbbf54a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e86eb251-mbhb4\" (UID: \"a29ded46-17c9-4189-8401-99218bbbf54a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86eb251-mbhb4" Oct 31 19:42:05.912627 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:05.912473 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29ded46-17c9-4189-8401-99218bbbf54a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e86eb251-mbhb4\" (UID: \"a29ded46-17c9-4189-8401-99218bbbf54a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86eb251-mbhb4" Oct 31 19:42:05.912874 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:05.912854 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dgmj\" (UniqueName: \"kubernetes.io/projected/a29ded46-17c9-4189-8401-99218bbbf54a-kube-api-access-8dgmj\") pod \"maas-api-revoke-sub-e86eb251-mbhb4\" (UID: \"a29ded46-17c9-4189-8401-99218bbbf54a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86eb251-mbhb4" Oct 31 19:42:06.007997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.007960 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:06.009082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.009052 4910 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 31 19:42:06.009215 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.009106 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" podUID="df617b70-0871-4423-b738-50476b66c833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 31 19:42:06.014031 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.014007 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dgmj\" (UniqueName: \"kubernetes.io/projected/a29ded46-17c9-4189-8401-99218bbbf54a-kube-api-access-8dgmj\") pod \"maas-api-revoke-sub-e86eb251-mbhb4\" (UID: \"a29ded46-17c9-4189-8401-99218bbbf54a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86eb251-mbhb4" Oct 31 19:42:06.014197 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.014171 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29ded46-17c9-4189-8401-99218bbbf54a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e86eb251-mbhb4\" (UID: \"a29ded46-17c9-4189-8401-99218bbbf54a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86eb251-mbhb4" Oct 31 19:42:06.014197 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.014194 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29ded46-17c9-4189-8401-99218bbbf54a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e86eb251-mbhb4\" (UID: \"a29ded46-17c9-4189-8401-99218bbbf54a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86eb251-mbhb4" Oct 31 19:42:06.014878 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.014855 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29ded46-17c9-4189-8401-99218bbbf54a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e86eb251-mbhb4\" (UID: \"a29ded46-17c9-4189-8401-99218bbbf54a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86eb251-mbhb4" Oct 31 19:42:06.016629 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.016608 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29ded46-17c9-4189-8401-99218bbbf54a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e86eb251-mbhb4\" (UID: \"a29ded46-17c9-4189-8401-99218bbbf54a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86eb251-mbhb4" Oct 31 19:42:06.027672 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.027607 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dgmj\" (UniqueName: \"kubernetes.io/projected/a29ded46-17c9-4189-8401-99218bbbf54a-kube-api-access-8dgmj\") pod \"maas-api-revoke-sub-e86eb251-mbhb4\" (UID: \"a29ded46-17c9-4189-8401-99218bbbf54a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86eb251-mbhb4" Oct 31 19:42:06.194296 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.194264 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86eb251-mbhb4" Oct 31 19:42:06.479477 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.479440 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4"] Oct 31 19:42:06.502029 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.495184 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4" Oct 31 19:42:06.502029 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.501280 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4"] Oct 31 19:42:06.528424 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.528330 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01593536-a296-4333-a1a2-cca5cedeb8f9-metrics-certs\") pod \"maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4\" (UID: \"01593536-a296-4333-a1a2-cca5cedeb8f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4" Oct 31 19:42:06.529262 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.528610 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01593536-a296-4333-a1a2-cca5cedeb8f9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4\" (UID: \"01593536-a296-4333-a1a2-cca5cedeb8f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4" Oct 31 19:42:06.529262 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.528971 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01593536-a296-4333-a1a2-cca5cedeb8f9-maas-api-tls\") pod \"maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4\" (UID: \"01593536-a296-4333-a1a2-cca5cedeb8f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4" Oct 31 19:42:06.529262 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.529000 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdqlw\" (UniqueName: \"kubernetes.io/projected/01593536-a296-4333-a1a2-cca5cedeb8f9-kube-api-access-hdqlw\") pod \"maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4\" (UID: \"01593536-a296-4333-a1a2-cca5cedeb8f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4" Oct 31 19:42:06.578111 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.578066 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a4e312-serving-cert\"" Oct 31 19:42:06.586399 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:42:06.586356 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29ded46_17c9_4189_8401_99218bbbf54a.slice/crio-8f3f049b5cd928727152234547c95b6b55f466d074ef04fd01fe4b834910306e WatchSource:0}: Error finding container 8f3f049b5cd928727152234547c95b6b55f466d074ef04fd01fe4b834910306e: Status 404 returned error can't find the container with id 8f3f049b5cd928727152234547c95b6b55f466d074ef04fd01fe4b834910306e Oct 31 19:42:06.587361 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.587331 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e86eb251-mbhb4"] Oct 31 19:42:06.635680 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.632752 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01593536-a296-4333-a1a2-cca5cedeb8f9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4\" (UID: \"01593536-a296-4333-a1a2-cca5cedeb8f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4" Oct 31 19:42:06.635680 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.633093 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01593536-a296-4333-a1a2-cca5cedeb8f9-maas-api-tls\") pod \"maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4\" (UID: \"01593536-a296-4333-a1a2-cca5cedeb8f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4" Oct 31 19:42:06.635680 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.633124 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdqlw\" (UniqueName: \"kubernetes.io/projected/01593536-a296-4333-a1a2-cca5cedeb8f9-kube-api-access-hdqlw\") pod \"maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4\" (UID: \"01593536-a296-4333-a1a2-cca5cedeb8f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4" Oct 31 19:42:06.635680 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.633236 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01593536-a296-4333-a1a2-cca5cedeb8f9-metrics-certs\") pod \"maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4\" (UID: \"01593536-a296-4333-a1a2-cca5cedeb8f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4" Oct 31 19:42:06.635680 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.635451 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01593536-a296-4333-a1a2-cca5cedeb8f9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4\" (UID: \"01593536-a296-4333-a1a2-cca5cedeb8f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4" Oct 31 19:42:06.657167 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.656951 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01593536-a296-4333-a1a2-cca5cedeb8f9-maas-api-tls\") pod \"maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4\" (UID: \"01593536-a296-4333-a1a2-cca5cedeb8f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4" Oct 31 19:42:06.658603 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.658550 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdqlw\" (UniqueName: \"kubernetes.io/projected/01593536-a296-4333-a1a2-cca5cedeb8f9-kube-api-access-hdqlw\") pod \"maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4\" (UID: \"01593536-a296-4333-a1a2-cca5cedeb8f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4" Oct 31 19:42:06.659017 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.658991 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01593536-a296-4333-a1a2-cca5cedeb8f9-metrics-certs\") pod \"maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4\" (UID: \"01593536-a296-4333-a1a2-cca5cedeb8f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4" Oct 31 19:42:06.863936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.863886 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4" Oct 31 19:42:06.889651 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:06.889562 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86eb251-mbhb4" event={"ID":"a29ded46-17c9-4189-8401-99218bbbf54a","Type":"ContainerStarted","Data":"8f3f049b5cd928727152234547c95b6b55f466d074ef04fd01fe4b834910306e"} Oct 31 19:42:07.015931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:07.009528 4910 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 31 19:42:07.015931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:07.009595 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" podUID="df617b70-0871-4423-b738-50476b66c833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 31 19:42:07.493849 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:07.493812 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4"] Oct 31 19:42:07.505870 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:42:07.505841 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01593536_a296_4333_a1a2_cca5cedeb8f9.slice/crio-428a4a377c21a9855b0ede8af053398c59b0d969d687e3492294a0650247e4f4 WatchSource:0}: Error finding container 428a4a377c21a9855b0ede8af053398c59b0d969d687e3492294a0650247e4f4: Status 404 returned error can't find the container with id 428a4a377c21a9855b0ede8af053398c59b0d969d687e3492294a0650247e4f4 Oct 31 19:42:07.903165 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:07.903121 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86eb251-mbhb4" event={"ID":"a29ded46-17c9-4189-8401-99218bbbf54a","Type":"ContainerStarted","Data":"694188f399057fff8517fe9525ad3d10c51ca5aac7137f6126c6dabc070dcc52"} Oct 31 19:42:07.905410 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:07.905384 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4" event={"ID":"01593536-a296-4333-a1a2-cca5cedeb8f9","Type":"ContainerStarted","Data":"428a4a377c21a9855b0ede8af053398c59b0d969d687e3492294a0650247e4f4"} Oct 31 19:42:07.938933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:07.935810 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86eb251-mbhb4" podStartSLOduration=2.93579218 podStartE2EDuration="2.93579218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:07.930357095 +0000 UTC m=+1362.293177467" watchObservedRunningTime="2026-10-31 19:42:07.93579218 +0000 UTC m=+1362.298612553" Oct 31 19:42:08.007846 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:08.007815 4910 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 31 19:42:08.007969 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:08.007878 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" podUID="df617b70-0871-4423-b738-50476b66c833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 31 19:42:08.942678 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:08.928347 4910 generic.go:356] "Generic (PLEG): container finished" podID="a29ded46-17c9-4189-8401-99218bbbf54a" containerID="694188f399057fff8517fe9525ad3d10c51ca5aac7137f6126c6dabc070dcc52" exitCode=0 Oct 31 19:42:08.942678 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:08.928971 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86eb251-mbhb4" event={"ID":"a29ded46-17c9-4189-8401-99218bbbf54a","Type":"ContainerDied","Data":"694188f399057fff8517fe9525ad3d10c51ca5aac7137f6126c6dabc070dcc52"} Oct 31 19:42:08.942678 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:08.931677 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4" event={"ID":"01593536-a296-4333-a1a2-cca5cedeb8f9","Type":"ContainerStarted","Data":"b8adbe1c5e2ab4f141f43373af7c0affc7682f476e812bff205d349731ad9ca4"} Oct 31 19:42:08.942678 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:08.931992 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4" Oct 31 19:42:08.981304 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:08.981245 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4" podStartSLOduration=2.501983723 podStartE2EDuration="2.981228544s" totalImagesPullingTime="479.244821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:08.977292052 +0000 UTC m=+1363.340112424" watchObservedRunningTime="2026-10-31 19:42:08.981228544 +0000 UTC m=+1363.344048917" Oct 31 19:42:09.008057 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:09.008015 4910 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 31 19:42:09.008228 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:09.008090 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" podUID="df617b70-0871-4423-b738-50476b66c833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 31 19:42:10.008132 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:10.008082 4910 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 31 19:42:10.008546 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:10.008155 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" podUID="df617b70-0871-4423-b738-50476b66c833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 31 19:42:10.406946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:10.393123 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86eb251-mbhb4" Oct 31 19:42:10.562997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:10.547405 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29ded46-17c9-4189-8401-99218bbbf54a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29ded46-17c9-4189-8401-99218bbbf54a-maas-api-openshift-ca\") pod \"a29ded46-17c9-4189-8401-99218bbbf54a\" (UID: \"a29ded46-17c9-4189-8401-99218bbbf54a\") " Oct 31 19:42:10.562997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:10.547743 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29ded46-17c9-4189-8401-99218bbbf54a-kube-api-access-8dgmj\" (UniqueName: \"kubernetes.io/projected/a29ded46-17c9-4189-8401-99218bbbf54a-kube-api-access-8dgmj\") pod \"a29ded46-17c9-4189-8401-99218bbbf54a\" (UID: \"a29ded46-17c9-4189-8401-99218bbbf54a\") " Oct 31 19:42:10.562997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:10.547839 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a29ded46-17c9-4189-8401-99218bbbf54a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29ded46-17c9-4189-8401-99218bbbf54a-maas-api-xks-ca\") pod \"a29ded46-17c9-4189-8401-99218bbbf54a\" (UID: \"a29ded46-17c9-4189-8401-99218bbbf54a\") " Oct 31 19:42:10.563688 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:10.563454 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29ded46-17c9-4189-8401-99218bbbf54a-maas-api-xks-ca" pod "a29ded46-17c9-4189-8401-99218bbbf54a" (UID: "a29ded46-17c9-4189-8401-99218bbbf54a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:42:10.564293 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:10.563968 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29ded46-17c9-4189-8401-99218bbbf54a-maas-api-openshift-ca" pod "a29ded46-17c9-4189-8401-99218bbbf54a" (UID: "a29ded46-17c9-4189-8401-99218bbbf54a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:42:10.567423 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:10.567316 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29ded46-17c9-4189-8401-99218bbbf54a-kube-api-access-8dgmj" pod "a29ded46-17c9-4189-8401-99218bbbf54a" (UID: "a29ded46-17c9-4189-8401-99218bbbf54a"). InnerVolumeSpecName "kube-api-access-8dgmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:42:10.651542 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:10.651461 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dgmj\" (UniqueName: \"kubernetes.io/projected/a29ded46-17c9-4189-8401-99218bbbf54a-kube-api-access-8dgmj\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:10.651542 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:10.651499 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29ded46-17c9-4189-8401-99218bbbf54a-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:10.651542 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:10.651514 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29ded46-17c9-4189-8401-99218bbbf54a-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:10.977148 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:10.977043 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86eb251-mbhb4" event={"ID":"a29ded46-17c9-4189-8401-99218bbbf54a","Type":"ContainerDied","Data":"8f3f049b5cd928727152234547c95b6b55f466d074ef04fd01fe4b834910306e"} Oct 31 19:42:10.977148 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:10.977059 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86eb251-mbhb4" Oct 31 19:42:10.977148 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:10.977093 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f3f049b5cd928727152234547c95b6b55f466d074ef04fd01fe4b834910306e" Oct 31 19:42:11.009378 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:11.007799 4910 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 31 19:42:11.009378 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:11.007901 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" podUID="df617b70-0871-4423-b738-50476b66c833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 31 19:42:12.009211 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:12.008480 4910 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Oct 31 19:42:12.009211 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:12.008550 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" podUID="df617b70-0871-4423-b738-50476b66c833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Oct 31 19:42:12.723417 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:42:12.723374 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:42:13.014479 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:13.014438 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:13.036129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:13.015021 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:13.036129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:13.016028 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:14.100388 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:14.100358 4910 generic.go:356] "Generic (PLEG): container finished" podID="706169ef-5dc5-4533-86af-fd02de456ab0" containerID="c88f574089b8579100c712a86a00aa52ee8ae7808ac21f7ef64c261a98cbcff8" exitCode=0 Oct 31 19:42:14.100818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:14.100436 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" event={"ID":"706169ef-5dc5-4533-86af-fd02de456ab0","Type":"ContainerDied","Data":"c88f574089b8579100c712a86a00aa52ee8ae7808ac21f7ef64c261a98cbcff8"} Oct 31 19:42:14.101184 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:14.101168 4910 scope.go:122] "RemoveContainer" containerID="c88f574089b8579100c712a86a00aa52ee8ae7808ac21f7ef64c261a98cbcff8" Oct 31 19:42:14.102983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:14.102954 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:42:14.103083 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:14.103029 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:42:14.103083 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:14.103062 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:42:14.964803 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:14.964768 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4" Oct 31 19:42:15.164259 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:15.164223 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" event={"ID":"706169ef-5dc5-4533-86af-fd02de456ab0","Type":"ContainerStarted","Data":"cc37f5426fae5d07363bc461faf8a4d307bc71f4bdb50f30f785a298622423b0"} Oct 31 19:42:15.288287 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:15.288175 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h"] Oct 31 19:42:15.290848 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:15.290814 4910 cpu_manager.go:414] "containerMap: removing container" podUID="a29ded46-17c9-4189-8401-99218bbbf54a" containerName="revoke-keys" Oct 31 19:42:15.290848 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:15.290848 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29ded46-17c9-4189-8401-99218bbbf54a" containerName="revoke-keys" Oct 31 19:42:15.291163 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:15.291082 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29ded46-17c9-4189-8401-99218bbbf54a" containerName="revoke-keys" Oct 31 19:42:15.295790 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:15.295764 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h" Oct 31 19:42:15.307534 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:15.306890 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h"] Oct 31 19:42:15.311455 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:15.311196 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-278c04-serving-cert\"" Oct 31 19:42:15.363977 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:15.363889 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/329c14cb-5f70-4dc3-996e-71429c09d2ae-maas-api-tls\") pod \"maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h\" (UID: \"329c14cb-5f70-4dc3-996e-71429c09d2ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h" Oct 31 19:42:15.364174 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:15.364015 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/329c14cb-5f70-4dc3-996e-71429c09d2ae-openshift-service-ca\") pod \"maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h\" (UID: \"329c14cb-5f70-4dc3-996e-71429c09d2ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h" Oct 31 19:42:15.364174 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:15.364047 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/329c14cb-5f70-4dc3-996e-71429c09d2ae-metrics-certs\") pod \"maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h\" (UID: \"329c14cb-5f70-4dc3-996e-71429c09d2ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h" Oct 31 19:42:15.364371 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:15.364329 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvck8\" (UniqueName: \"kubernetes.io/projected/329c14cb-5f70-4dc3-996e-71429c09d2ae-kube-api-access-bvck8\") pod \"maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h\" (UID: \"329c14cb-5f70-4dc3-996e-71429c09d2ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h" Oct 31 19:42:15.465824 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:15.465786 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvck8\" (UniqueName: \"kubernetes.io/projected/329c14cb-5f70-4dc3-996e-71429c09d2ae-kube-api-access-bvck8\") pod \"maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h\" (UID: \"329c14cb-5f70-4dc3-996e-71429c09d2ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h" Oct 31 19:42:15.466039 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:15.465994 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/329c14cb-5f70-4dc3-996e-71429c09d2ae-maas-api-tls\") pod \"maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h\" (UID: \"329c14cb-5f70-4dc3-996e-71429c09d2ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h" Oct 31 19:42:15.466995 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:15.466967 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/329c14cb-5f70-4dc3-996e-71429c09d2ae-openshift-service-ca\") pod \"maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h\" (UID: \"329c14cb-5f70-4dc3-996e-71429c09d2ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h" Oct 31 19:42:15.467139 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:15.467004 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/329c14cb-5f70-4dc3-996e-71429c09d2ae-metrics-certs\") pod \"maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h\" (UID: \"329c14cb-5f70-4dc3-996e-71429c09d2ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h" Oct 31 19:42:15.468126 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:15.468101 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/329c14cb-5f70-4dc3-996e-71429c09d2ae-openshift-service-ca\") pod \"maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h\" (UID: \"329c14cb-5f70-4dc3-996e-71429c09d2ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h" Oct 31 19:42:15.469239 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:15.468896 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/329c14cb-5f70-4dc3-996e-71429c09d2ae-maas-api-tls\") pod \"maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h\" (UID: \"329c14cb-5f70-4dc3-996e-71429c09d2ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h" Oct 31 19:42:15.480601 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:15.480579 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/329c14cb-5f70-4dc3-996e-71429c09d2ae-metrics-certs\") pod \"maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h\" (UID: \"329c14cb-5f70-4dc3-996e-71429c09d2ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h" Oct 31 19:42:15.481740 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:15.481719 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvck8\" (UniqueName: \"kubernetes.io/projected/329c14cb-5f70-4dc3-996e-71429c09d2ae-kube-api-access-bvck8\") pod \"maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h\" (UID: \"329c14cb-5f70-4dc3-996e-71429c09d2ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h" Oct 31 19:42:15.661939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:15.656229 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h" Oct 31 19:42:15.661939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:15.656715 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:42:15.665968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:15.665069 4910 patch_prober.go:29] interesting pod/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 19:42:15.665968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:15.665184 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" podUID="706169ef-5dc5-4533-86af-fd02de456ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 19:42:15.893956 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:15.887433 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:42:16.674280 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:16.674244 4910 patch_prober.go:29] interesting pod/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 19:42:16.674889 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:16.674853 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" podUID="706169ef-5dc5-4533-86af-fd02de456ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 19:42:16.774018 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:16.770466 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h"] Oct 31 19:42:16.848937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:16.847015 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq"] Oct 31 19:42:16.879944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:16.877109 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq"] Oct 31 19:42:16.879944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:16.877237 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:16.959038 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:16.949218 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b1fae7-openshift-default-dockercfg-nx28t\"" Oct 31 19:42:16.971946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:16.971136 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-workload-certs\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:16.971946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:16.971254 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-credential-socket\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:16.971946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:16.971341 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-token\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:16.971946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:16.971449 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhm4w\" (UniqueName: \"kubernetes.io/projected/9f646f52-3c80-4fbb-8049-712f1e6b1595-kube-api-access-nhm4w\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:16.971946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:16.971522 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-workload-socket\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:16.971946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:16.971604 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-envoy\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:16.971946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:16.971657 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-data\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:16.988934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:16.980069 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-podinfo\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:16.988934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:16.980232 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f646f52-3c80-4fbb-8049-712f1e6b1595-istiod-ca-cert\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:17.210038 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:17.209268 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-podinfo\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:17.210038 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:17.209338 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f646f52-3c80-4fbb-8049-712f1e6b1595-istiod-ca-cert\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:17.210038 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:17.209437 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-workload-certs\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:17.210038 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:17.209489 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-credential-socket\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:17.210038 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:17.209558 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-token\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:17.210038 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:17.209637 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhm4w\" (UniqueName: \"kubernetes.io/projected/9f646f52-3c80-4fbb-8049-712f1e6b1595-kube-api-access-nhm4w\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:17.210038 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:17.209689 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-workload-socket\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:17.210038 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:17.209726 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-envoy\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:17.210038 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:17.209765 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-data\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:17.210614 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:17.210472 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-credential-socket\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:17.222698 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:17.211751 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-data\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:17.222698 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:17.220073 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f646f52-3c80-4fbb-8049-712f1e6b1595-istiod-ca-cert\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:17.222698 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:17.220407 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-workload-certs\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:17.222698 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:17.220646 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-workload-socket\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:17.421101 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:17.398247 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhm4w\" (UniqueName: \"kubernetes.io/projected/9f646f52-3c80-4fbb-8049-712f1e6b1595-kube-api-access-nhm4w\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:17.421101 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:17.403186 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-token\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:17.421101 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:17.420563 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-envoy\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:17.506355 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:17.504788 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-podinfo\") pod \"e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:17.575948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:17.575888 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h" event={"ID":"329c14cb-5f70-4dc3-996e-71429c09d2ae","Type":"ContainerStarted","Data":"046e89a9c4737dff29ca62a6177bbcf944d48f3fe08fdb36472e09df9485a0fc"} Oct 31 19:42:17.662356 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:17.662323 4910 patch_prober.go:29] interesting pod/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 19:42:17.662620 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:17.662594 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" podUID="706169ef-5dc5-4533-86af-fd02de456ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 19:42:17.704553 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:17.704524 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:18.439985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:18.425425 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq"] Oct 31 19:42:18.476624 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:42:18.474012 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f646f52_3c80_4fbb_8049_712f1e6b1595.slice/crio-41121d7f2e10a55856058b531afc3c956cd5d5aca34faef975214bc6f0b29cb7 WatchSource:0}: Error finding container 41121d7f2e10a55856058b531afc3c956cd5d5aca34faef975214bc6f0b29cb7: Status 404 returned error can't find the container with id 41121d7f2e10a55856058b531afc3c956cd5d5aca34faef975214bc6f0b29cb7 Oct 31 19:42:18.480107 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:18.479652 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:42:18.480107 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:18.479799 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:42:18.480107 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:18.479884 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:42:18.601627 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:18.601458 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h" event={"ID":"329c14cb-5f70-4dc3-996e-71429c09d2ae","Type":"ContainerStarted","Data":"9a9a7334279751eabe032b67c8985843aae4135d7c632a3309cfc5a05862c839"} Oct 31 19:42:18.601627 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:18.601591 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h" Oct 31 19:42:18.604941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:18.603953 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" event={"ID":"9f646f52-3c80-4fbb-8049-712f1e6b1595","Type":"ContainerStarted","Data":"41121d7f2e10a55856058b531afc3c956cd5d5aca34faef975214bc6f0b29cb7"} Oct 31 19:42:18.630940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:18.629620 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h" podStartSLOduration=2.898670424 podStartE2EDuration="3.629599976s" totalImagesPullingTime="730.929552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:18.627413059 +0000 UTC m=+1372.990233432" watchObservedRunningTime="2026-10-31 19:42:18.629599976 +0000 UTC m=+1372.992420349" Oct 31 19:42:18.666718 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:18.666354 4910 patch_prober.go:29] interesting pod/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 19:42:18.666718 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:18.666425 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" podUID="706169ef-5dc5-4533-86af-fd02de456ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 19:42:19.664942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:19.663042 4910 patch_prober.go:29] interesting pod/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 19:42:19.664942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:19.663106 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" podUID="706169ef-5dc5-4533-86af-fd02de456ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 19:42:19.675957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:19.672638 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" event={"ID":"9f646f52-3c80-4fbb-8049-712f1e6b1595","Type":"ContainerStarted","Data":"6f3b83ca6f6ad605de24d7751c2606b03ebbe6624508c6c6865d37a6e9523cbb"} Oct 31 19:42:19.731940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:19.722128 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:19.767481 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:19.764349 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" podStartSLOduration=3.764325748 podStartE2EDuration="3.764325748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:19.722583482 +0000 UTC m=+1374.085403855" watchObservedRunningTime="2026-10-31 19:42:19.764325748 +0000 UTC m=+1374.127146129" Oct 31 19:42:19.829052 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:19.823680 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:20.653988 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:20.653136 4910 patch_prober.go:29] interesting pod/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Oct 31 19:42:20.653988 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:20.653197 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" podUID="706169ef-5dc5-4533-86af-fd02de456ab0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Oct 31 19:42:20.682871 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:20.682838 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:20.684025 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:20.684000 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:42:21.865857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:21.865826 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:42:21.868663 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:21.868639 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:42:24.679414 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:24.679387 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h" Oct 31 19:42:27.031347 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:27.031312 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeb94931-j9mpm"] Oct 31 19:42:27.036118 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:27.036096 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeb94931-j9mpm"] Oct 31 19:42:27.759755 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:27.759720 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38b16441-a7cb-4418-9306-13d4c448f7d4" path="/var/lib/kubelet/pods/38b16441-a7cb-4418-9306-13d4c448f7d4/volumes" Oct 31 19:42:29.521826 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:29.521789 4910 scope.go:122] "RemoveContainer" containerID="30e21a4855c8be96cb1e0e5db2b175df59848fc5f3e669793624eaf492d64936" Oct 31 19:42:29.573255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:29.573229 4910 scope.go:122] "RemoveContainer" containerID="6ebc6bfd83c60d818d2100ed6cfe63fde9445d4960c68b3929217077289a31f3" Oct 31 19:42:37.848114 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:37.842535 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e73256-5lm7h"] Oct 31 19:42:37.887193 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:37.887157 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e73256-5lm7h"] Oct 31 19:42:39.758876 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:39.758837 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a480bcfb-f234-4e55-a2fb-15990c28a0a1" path="/var/lib/kubelet/pods/a480bcfb-f234-4e55-a2fb-15990c28a0a1/volumes" Oct 31 19:42:43.138998 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:42:43.138962 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:42:44.920937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:44.920163 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:44.923130 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:44.923094 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": context deadline exceeded" Oct 31 19:42:45.073308 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:45.072022 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:45.228987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:45.226938 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:42:45.521119 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:45.521015 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:45.771999 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:45.763042 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:42:45.771999 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:45.765862 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:45.771999 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:45.766173 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:42:45.771999 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:45.766453 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 19:42:45.774064 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:45.774031 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:42:45.799942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:45.796605 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:45.960935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:45.922805 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:45.983069 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:45.983036 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:42:46.004450 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:46.004014 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6qm4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:42:46.004450 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:46.004079 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" podUID="c73a891d-f2de-4099-9a98-bdcbae51888f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:46.074561 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:46.074519 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:42:46.218109 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:46.218027 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:46.308019 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:46.238609 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:42:46.333309 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:46.333215 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:42:46.503849 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:46.503771 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk"] Oct 31 19:42:46.505952 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:46.505897 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" podUID="507b2fb7-a22a-4cac-9d4a-23199f7fcbff" containerName="istio-proxy" containerID="cri-o://0ee1fe21413c867adee167d9aac19f036cf7a8de9dc3f7c257ca2d48b0d929f5" gracePeriod=30 Oct 31 19:42:46.522238 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:46.522197 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:46.532329 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:46.532294 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:42:46.638847 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:46.638755 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p"] Oct 31 19:42:46.651714 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:46.651672 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" podUID="df617b70-0871-4423-b738-50476b66c833" containerName="istio-proxy" containerID="cri-o://508db727dd9f175dee66c2f1aa91cf0cbb554a09f6947c83a70ef71c9bca48e5" gracePeriod=30 Oct 31 19:42:46.782289 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:46.782236 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:42:46.784692 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:46.784662 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:42:46.837420 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:46.837366 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:46.848471 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:46.840724 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:42:46.931992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:46.893700 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:47.026262 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:47.026214 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:42:47.054550 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:47.054509 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:47.461937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:47.075277 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:47.461937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:47.220547 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded" Oct 31 19:42:47.461937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:47.221934 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:47.461937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:47.222457 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:47.461937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:47.397009 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq"] Oct 31 19:42:47.461937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:47.430041 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:42:47.461937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:47.430728 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" podUID="00f5bd2c-c4f2-4854-8169-dd65ad5d301d" containerName="istio-proxy" containerID="cri-o://21f020cabaea277be7fcf2cdd96320e702873f241f117e4ee74ccddc0ca4fc4f" gracePeriod=30 Oct 31 19:42:47.626988 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:47.618783 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:47.834058 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:47.834005 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:47.951246 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:47.870109 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:47.951246 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:47.853034 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:47.951246 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:47.834118 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:42:47.951246 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:47.905094 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:42:47.951246 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:47.907080 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:42:47.951246 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:47.925387 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:42:48.008445 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:48.007055 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:48.008445 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:48.007467 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:48.160989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:48.160647 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:42:48.383493 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:48.380872 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:48.388046 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:48.387991 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:48.501073 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:48.410874 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:42:48.501073 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:48.411516 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:42:48.662661 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:48.662616 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": context deadline exceeded" Oct 31 19:42:48.796502 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:48.796391 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:48.868847 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:48.859227 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:48.898936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:48.881174 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:42:48.898936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:48.882363 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:42:48.898936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:48.882610 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:42:48.898936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:48.882795 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:42:48.898936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:48.883380 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:42:48.985901 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:48.985860 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:42:48.996334 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:48.986204 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:42:49.014285 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:48.992369 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:49.014285 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:48.992404 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:42:49.014285 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.012133 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:42:49.040194 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.023713 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:49.040194 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.026711 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:49.040194 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.027784 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:42:49.111998 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.105530 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7"] Oct 31 19:42:49.158952 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.158703 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:42:49.159146 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.158970 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7" Oct 31 19:42:49.286210 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.286169 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f403772-348e-44cb-adf1-c695cf83f1ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7\" (UID: \"7f403772-348e-44cb-adf1-c695cf83f1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7" Oct 31 19:42:49.304631 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.304595 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fv98\" (UniqueName: \"kubernetes.io/projected/7f403772-348e-44cb-adf1-c695cf83f1ee-kube-api-access-2fv98\") pod \"maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7\" (UID: \"7f403772-348e-44cb-adf1-c695cf83f1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7" Oct 31 19:42:49.305104 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.305076 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f403772-348e-44cb-adf1-c695cf83f1ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7\" (UID: \"7f403772-348e-44cb-adf1-c695cf83f1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7" Oct 31 19:42:49.341957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.317023 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:42:49.359038 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.320249 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:42:49.396689 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.396604 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7"] Oct 31 19:42:49.435151 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.418402 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:49.456617 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.445841 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f403772-348e-44cb-adf1-c695cf83f1ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7\" (UID: \"7f403772-348e-44cb-adf1-c695cf83f1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7" Oct 31 19:42:49.456617 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.446262 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f403772-348e-44cb-adf1-c695cf83f1ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7\" (UID: \"7f403772-348e-44cb-adf1-c695cf83f1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7" Oct 31 19:42:49.456617 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.446322 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fv98\" (UniqueName: \"kubernetes.io/projected/7f403772-348e-44cb-adf1-c695cf83f1ee-kube-api-access-2fv98\") pod \"maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7\" (UID: \"7f403772-348e-44cb-adf1-c695cf83f1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7" Oct 31 19:42:49.457181 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.457146 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:49.486431 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.486380 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:49.497207 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.497171 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f403772-348e-44cb-adf1-c695cf83f1ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7\" (UID: \"7f403772-348e-44cb-adf1-c695cf83f1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7" Oct 31 19:42:49.526973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.524420 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f403772-348e-44cb-adf1-c695cf83f1ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7\" (UID: \"7f403772-348e-44cb-adf1-c695cf83f1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7" Oct 31 19:42:49.828285 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.797135 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fv98\" (UniqueName: \"kubernetes.io/projected/7f403772-348e-44cb-adf1-c695cf83f1ee-kube-api-access-2fv98\") pod \"maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7\" (UID: \"7f403772-348e-44cb-adf1-c695cf83f1ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7" Oct 31 19:42:49.921018 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.920974 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:42:49.921557 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.921525 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:49.921992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.921965 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:42:49.961002 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.960959 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7" Oct 31 19:42:49.998759 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.998715 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:50.000303 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:49.999026 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:50.034286 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:50.025481 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:50.034286 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:50.025847 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:42:50.034286 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:50.026094 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:42:50.046582 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:50.026982 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:42:50.100270 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:50.100173 4910 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:42:50.100551 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:50.100517 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" podUID="00f5bd2c-c4f2-4854-8169-dd65ad5d301d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:42:50.528997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:50.527895 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Oct 31 19:42:50.583001 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:50.560203 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:42:50.584558 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:50.560255 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 19:42:50.586128 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:50.560278 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:42:50.981948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:50.961502 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:50.981948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:50.965434 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:50.981948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:50.967487 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:42:51.072397 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:51.072337 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:51.097640 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:51.097596 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:51.098234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:51.098187 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:42:51.098627 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:51.098605 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:42:51.180466 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:51.178044 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:51.189879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:51.189840 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:42:51.454933 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:42:51.452652 4910 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.366s" Oct 31 19:42:51.454933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:51.453090 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:42:51.483196 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:51.483159 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:42:51.800295 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:51.798305 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:51.800295 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:51.798773 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:42:52.018997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:52.017120 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:42:52.121104 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:52.121057 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:42:52.244240 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:52.244180 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:52.387940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:52.363291 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:42:52.426352 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:52.364359 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:42:52.540939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:52.364609 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:42:52.540939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:52.410150 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:52.540939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:52.411579 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": context deadline exceeded" Oct 31 19:42:52.540939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:52.474293 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:42:52.540939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:52.497826 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:42:52.594155 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:52.578540 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:42:52.854138 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:52.854043 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:52.874080 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:52.863426 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:53.027634 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.020835 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:53.054208 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.054168 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:42:53.126488 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.125891 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:53.268333 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.268074 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:42:53.317503 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.317465 4910 generic.go:356] "Generic (PLEG): container finished" podID="00f5bd2c-c4f2-4854-8169-dd65ad5d301d" containerID="21f020cabaea277be7fcf2cdd96320e702873f241f117e4ee74ccddc0ca4fc4f" exitCode=0 Oct 31 19:42:53.318066 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.318036 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" event={"ID":"00f5bd2c-c4f2-4854-8169-dd65ad5d301d","Type":"ContainerDied","Data":"21f020cabaea277be7fcf2cdd96320e702873f241f117e4ee74ccddc0ca4fc4f"} Oct 31 19:42:53.364288 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.364254 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb"] Oct 31 19:42:53.487366 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.461927 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:53.519856 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.519798 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:42:53.538326 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.538287 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:42:53.539618 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.539587 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:42:53.540763 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.540740 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:42:53.547931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.543980 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1e8cf0-openshift-default-dockercfg-g8d2s\"" Oct 31 19:42:53.559030 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.537242 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:42:53.631149 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.601355 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7" event={"ID":"7f403772-348e-44cb-adf1-c695cf83f1ee","Type":"ContainerStarted","Data":"701e920b48ece24cbf1890b02652142a6428627053316d968563aff4ed441ef5"} Oct 31 19:42:53.694130 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.694064 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:53.712617 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.712579 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb"] Oct 31 19:42:53.712978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.712943 4910 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Oct 31 19:42:53.723110 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.723071 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" podUID="507b2fb7-a22a-4cac-9d4a-23199f7fcbff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Oct 31 19:42:53.723318 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.713306 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:53.723727 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.712958 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:42:53.737704 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.729545 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:42:53.737704 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.730326 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:53.796647 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.791833 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vswhv\" (UniqueName: \"kubernetes.io/projected/c86e67dd-5de9-4213-b26d-fcbc3bd90353-kube-api-access-vswhv\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:53.796647 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.791951 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-envoy\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:53.796647 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.792015 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istiod-ca-cert\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:53.796647 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.792094 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-data\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:53.796647 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.792193 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-workload-certs\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:53.796647 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.792229 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-token\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:53.796647 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.792475 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-podinfo\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:53.796647 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.792517 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-credential-socket\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:53.796647 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.792645 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-workload-socket\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:53.870310 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.870199 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:53.925292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.925234 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-workload-socket\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:53.942522 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.942470 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vswhv\" (UniqueName: \"kubernetes.io/projected/c86e67dd-5de9-4213-b26d-fcbc3bd90353-kube-api-access-vswhv\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:53.966740 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.925897 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-workload-socket\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:53.967710 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.967224 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-envoy\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:53.967992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.967962 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istiod-ca-cert\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:53.993329 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.993238 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-data\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:53.993705 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.993676 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-workload-certs\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:53.994886 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.994854 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-token\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:54.002563 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:53.995348 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-podinfo\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:54.019943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.018187 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-credential-socket\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:54.024050 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.022150 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:42:54.213956 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.187161 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-workload-certs\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:54.213956 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.188333 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vswhv\" (UniqueName: \"kubernetes.io/projected/c86e67dd-5de9-4213-b26d-fcbc3bd90353-kube-api-access-vswhv\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:54.213956 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.188453 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-data\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:54.225942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.215103 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 19:42:54.226436 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.226406 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-credential-socket\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:54.359940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.359880 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:54.360979 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.360942 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:42:54.362636 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.362608 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istiod-ca-cert\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:54.383571 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.383499 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-envoy\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:54.408222 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.408183 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-podinfo\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:54.445258 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.445223 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-token\") pod \"e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:54.532657 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.532610 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:54.629991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.618227 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:42:54.629991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.618539 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded" Oct 31 19:42:54.629991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.618573 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:42:54.657805 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.657771 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:54.679104 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.679018 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:54.749494 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.745175 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:42:54.749494 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.745567 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:54.749494 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.745844 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:42:54.771959 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.750607 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:54.771959 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.763021 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:42:54.789975 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.789625 4910 generic.go:356] "Generic (PLEG): container finished" podID="df617b70-0871-4423-b738-50476b66c833" containerID="508db727dd9f175dee66c2f1aa91cf0cbb554a09f6947c83a70ef71c9bca48e5" exitCode=0 Oct 31 19:42:54.794085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.793768 4910 generic.go:356] "Generic (PLEG): container finished" podID="507b2fb7-a22a-4cac-9d4a-23199f7fcbff" containerID="0ee1fe21413c867adee167d9aac19f036cf7a8de9dc3f7c257ca2d48b0d929f5" exitCode=0 Oct 31 19:42:54.797131 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.796377 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" Oct 31 19:42:54.797213 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:42:54.797155 4910 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.049s" Oct 31 19:42:54.797213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.797205 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" event={"ID":"df617b70-0871-4423-b738-50476b66c833","Type":"ContainerDied","Data":"508db727dd9f175dee66c2f1aa91cf0cbb554a09f6947c83a70ef71c9bca48e5"} Oct 31 19:42:54.797330 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.797298 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:42:54.797526 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.797452 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" event={"ID":"507b2fb7-a22a-4cac-9d4a-23199f7fcbff","Type":"ContainerDied","Data":"0ee1fe21413c867adee167d9aac19f036cf7a8de9dc3f7c257ca2d48b0d929f5"} Oct 31 19:42:54.797631 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.797543 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:42:54.797631 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.797582 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq" event={"ID":"00f5bd2c-c4f2-4854-8169-dd65ad5d301d","Type":"ContainerDied","Data":"1a99dd743d0ec534887b3ff879094ffc5874eaf5d0c748cf93b0c4a916e59cea"} Oct 31 19:42:54.797754 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.797648 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:42:54.801184 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.801164 4910 scope.go:122] "RemoveContainer" containerID="8ecb05670fd8cd8251e58cd0e3bdb1a8db6549d99b0a85c8015e2a152950365c" Oct 31 19:42:54.891365 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.891276 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istiod-ca-cert\") pod \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " Oct 31 19:42:54.892025 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.891999 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-token\" (UniqueName: \"kubernetes.io/projected/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-token\") pod \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " Oct 31 19:42:54.906405 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.906373 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-workload-certs\") pod \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " Oct 31 19:42:54.906609 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.906591 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-podinfo\") pod \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " Oct 31 19:42:54.906777 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.906758 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-data\") pod \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " Oct 31 19:42:54.906985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.906968 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-envoy\") pod \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " Oct 31 19:42:54.907184 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.907167 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-workload-socket\") pod \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " Oct 31 19:42:54.907447 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.907429 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-kube-api-access-n5nxn\" (UniqueName: \"kubernetes.io/projected/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-kube-api-access-n5nxn\") pod \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " Oct 31 19:42:54.907653 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.907636 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-credential-socket\") pod \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\" (UID: \"00f5bd2c-c4f2-4854-8169-dd65ad5d301d\") " Oct 31 19:42:54.917348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.917236 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:42:54.933253 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.933213 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:54.933662 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.933635 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:54.934729 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.934701 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-workload-certs" pod "00f5bd2c-c4f2-4854-8169-dd65ad5d301d" (UID: "00f5bd2c-c4f2-4854-8169-dd65ad5d301d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:54.935766 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.935745 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istiod-ca-cert" pod "00f5bd2c-c4f2-4854-8169-dd65ad5d301d" (UID: "00f5bd2c-c4f2-4854-8169-dd65ad5d301d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:42:54.937351 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.937330 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-data" pod "00f5bd2c-c4f2-4854-8169-dd65ad5d301d" (UID: "00f5bd2c-c4f2-4854-8169-dd65ad5d301d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:54.975444 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:54.975110 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-workload-socket" pod "00f5bd2c-c4f2-4854-8169-dd65ad5d301d" (UID: "00f5bd2c-c4f2-4854-8169-dd65ad5d301d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:55.030326 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.030164 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:42:55.031506 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.031428 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:42:55.039403 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.039375 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-credential-socket" pod "00f5bd2c-c4f2-4854-8169-dd65ad5d301d" (UID: "00f5bd2c-c4f2-4854-8169-dd65ad5d301d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:55.065953 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.058879 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:55.094937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.070156 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:55.266154 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.227366 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:55.266154 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.227400 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:55.266154 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.227416 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:55.266154 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.227432 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:55.266154 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.169945 4910 scope.go:122] "RemoveContainer" containerID="21f020cabaea277be7fcf2cdd96320e702873f241f117e4ee74ccddc0ca4fc4f" Oct 31 19:42:55.266154 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.122136 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:42:55.293314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.278167 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-envoy" pod "00f5bd2c-c4f2-4854-8169-dd65ad5d301d" (UID: "00f5bd2c-c4f2-4854-8169-dd65ad5d301d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:55.359936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.311335 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-kube-api-access-n5nxn" pod "00f5bd2c-c4f2-4854-8169-dd65ad5d301d" (UID: "00f5bd2c-c4f2-4854-8169-dd65ad5d301d"). InnerVolumeSpecName "kube-api-access-n5nxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:42:55.360511 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.360482 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7"] Oct 31 19:42:55.371942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.362743 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5nxn\" (UniqueName: \"kubernetes.io/projected/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-kube-api-access-n5nxn\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:55.400765 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.399992 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:55.400765 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.383971 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:55.400765 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.383860 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:42:55.400765 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.400470 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:42:55.623171 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.623132 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-token" pod "00f5bd2c-c4f2-4854-8169-dd65ad5d301d" (UID: "00f5bd2c-c4f2-4854-8169-dd65ad5d301d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:42:55.637946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.634217 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:55.637946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.635513 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:55.637946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.636904 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:55.679814 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.539264 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:55.716724 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.716694 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:42:55.894761 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.894659 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-podinfo" pod "00f5bd2c-c4f2-4854-8169-dd65ad5d301d" (UID: "00f5bd2c-c4f2-4854-8169-dd65ad5d301d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:42:55.996449 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:55.955478 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00f5bd2c-c4f2-4854-8169-dd65ad5d301d-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:56.039956 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.036678 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:56.213330 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.213269 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:42:56.215363 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.215327 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:42:56.215596 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.215568 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:42:56.377541 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.374884 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:42:56.404391 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.403398 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:42:56.534155 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.507567 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:42:56.534155 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.516229 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:42:56.559398 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.555857 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk" Oct 31 19:42:56.559592 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.559561 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7" event={"ID":"7f403772-348e-44cb-adf1-c695cf83f1ee","Type":"ContainerStarted","Data":"8115cc157caf6bb6cebc5eb34257bac13cf78f45e1727245bd3ed3fa789de338"} Oct 31 19:42:56.628183 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.612947 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:42:56.681936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.680892 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-podinfo\") pod \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " Oct 31 19:42:56.681936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.680990 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-envoy\") pod \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " Oct 31 19:42:56.788164 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.765300 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-workload-socket\") pod \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " Oct 31 19:42:56.788164 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.765421 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-credential-socket\") pod \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " Oct 31 19:42:56.788164 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.765522 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-kube-api-access-psjfm\" (UniqueName: \"kubernetes.io/projected/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-kube-api-access-psjfm\") pod \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " Oct 31 19:42:56.788164 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.765563 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-data\") pod \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " Oct 31 19:42:56.788164 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.765662 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-workload-certs\") pod \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " Oct 31 19:42:56.788164 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.765715 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istiod-ca-cert\") pod \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " Oct 31 19:42:56.788164 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.765749 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-token\" (UniqueName: \"kubernetes.io/projected/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-token\") pod \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\" (UID: \"507b2fb7-a22a-4cac-9d4a-23199f7fcbff\") " Oct 31 19:42:56.788164 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.785043 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-workload-socket" pod "507b2fb7-a22a-4cac-9d4a-23199f7fcbff" (UID: "507b2fb7-a22a-4cac-9d4a-23199f7fcbff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:56.788164 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.785355 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:56.793581 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.793554 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istiod-ca-cert" pod "507b2fb7-a22a-4cac-9d4a-23199f7fcbff" (UID: "507b2fb7-a22a-4cac-9d4a-23199f7fcbff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:42:56.794005 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.793982 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-data" pod "507b2fb7-a22a-4cac-9d4a-23199f7fcbff" (UID: "507b2fb7-a22a-4cac-9d4a-23199f7fcbff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:56.794470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.794449 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-workload-certs" pod "507b2fb7-a22a-4cac-9d4a-23199f7fcbff" (UID: "507b2fb7-a22a-4cac-9d4a-23199f7fcbff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:56.795019 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.794993 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-credential-socket" pod "507b2fb7-a22a-4cac-9d4a-23199f7fcbff" (UID: "507b2fb7-a22a-4cac-9d4a-23199f7fcbff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:56.797893 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.797645 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-envoy" pod "507b2fb7-a22a-4cac-9d4a-23199f7fcbff" (UID: "507b2fb7-a22a-4cac-9d4a-23199f7fcbff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:56.798103 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.798031 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:42:56.798891 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.798868 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-kube-api-access-psjfm" pod "507b2fb7-a22a-4cac-9d4a-23199f7fcbff" (UID: "507b2fb7-a22a-4cac-9d4a-23199f7fcbff"). InnerVolumeSpecName "kube-api-access-psjfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:42:56.803833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.803810 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-token" pod "507b2fb7-a22a-4cac-9d4a-23199f7fcbff" (UID: "507b2fb7-a22a-4cac-9d4a-23199f7fcbff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:42:56.803977 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.803841 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-podinfo" pod "507b2fb7-a22a-4cac-9d4a-23199f7fcbff" (UID: "507b2fb7-a22a-4cac-9d4a-23199f7fcbff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:42:56.894014 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.889411 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-istio-data\") pod \"df617b70-0871-4423-b738-50476b66c833\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " Oct 31 19:42:56.894014 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.889499 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df617b70-0871-4423-b738-50476b66c833-kube-api-access-bzrbm\" (UniqueName: \"kubernetes.io/projected/df617b70-0871-4423-b738-50476b66c833-kube-api-access-bzrbm\") pod \"df617b70-0871-4423-b738-50476b66c833\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " Oct 31 19:42:56.894014 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.889668 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-workload-socket\") pod \"df617b70-0871-4423-b738-50476b66c833\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " Oct 31 19:42:56.894014 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.889848 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-workload-certs\") pod \"df617b70-0871-4423-b738-50476b66c833\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " Oct 31 19:42:56.894014 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.889882 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df617b70-0871-4423-b738-50476b66c833-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df617b70-0871-4423-b738-50476b66c833-istiod-ca-cert\") pod \"df617b70-0871-4423-b738-50476b66c833\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " Oct 31 19:42:56.894014 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.889952 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-credential-socket\") pod \"df617b70-0871-4423-b738-50476b66c833\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " Oct 31 19:42:56.894014 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.890000 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-istio-envoy\") pod \"df617b70-0871-4423-b738-50476b66c833\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " Oct 31 19:42:56.894014 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.890057 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df617b70-0871-4423-b738-50476b66c833-istio-token\" (UniqueName: \"kubernetes.io/projected/df617b70-0871-4423-b738-50476b66c833-istio-token\") pod \"df617b70-0871-4423-b738-50476b66c833\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " Oct 31 19:42:56.894014 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.890172 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df617b70-0871-4423-b738-50476b66c833-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df617b70-0871-4423-b738-50476b66c833-istio-podinfo\") pod \"df617b70-0871-4423-b738-50476b66c833\" (UID: \"df617b70-0871-4423-b738-50476b66c833\") " Oct 31 19:42:56.894653 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.894405 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:56.894653 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.894432 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:56.894653 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.894448 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:56.894653 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.894463 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:56.894653 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.894478 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:56.894653 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.894495 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:56.894653 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.894509 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:56.894653 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.894524 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psjfm\" (UniqueName: \"kubernetes.io/projected/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-kube-api-access-psjfm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:56.894653 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.894539 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/507b2fb7-a22a-4cac-9d4a-23199f7fcbff-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:56.901755 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.901513 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-istio-data" pod "df617b70-0871-4423-b738-50476b66c833" (UID: "df617b70-0871-4423-b738-50476b66c833"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:56.904031 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.904003 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df617b70-0871-4423-b738-50476b66c833-istiod-ca-cert" pod "df617b70-0871-4423-b738-50476b66c833" (UID: "df617b70-0871-4423-b738-50476b66c833"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:42:56.905271 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.905076 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-workload-socket" pod "df617b70-0871-4423-b738-50476b66c833" (UID: "df617b70-0871-4423-b738-50476b66c833"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:56.905688 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.905615 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-credential-socket" pod "df617b70-0871-4423-b738-50476b66c833" (UID: "df617b70-0871-4423-b738-50476b66c833"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:56.905954 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.905858 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-workload-certs" pod "df617b70-0871-4423-b738-50476b66c833" (UID: "df617b70-0871-4423-b738-50476b66c833"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:56.909094 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.907876 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df617b70-0871-4423-b738-50476b66c833-istio-podinfo" pod "df617b70-0871-4423-b738-50476b66c833" (UID: "df617b70-0871-4423-b738-50476b66c833"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:42:56.911080 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.911044 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-istio-envoy" pod "df617b70-0871-4423-b738-50476b66c833" (UID: "df617b70-0871-4423-b738-50476b66c833"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:42:56.911189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.911108 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df617b70-0871-4423-b738-50476b66c833-istio-token" pod "df617b70-0871-4423-b738-50476b66c833" (UID: "df617b70-0871-4423-b738-50476b66c833"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:42:56.915083 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:56.912756 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df617b70-0871-4423-b738-50476b66c833-kube-api-access-bzrbm" pod "df617b70-0871-4423-b738-50476b66c833" (UID: "df617b70-0871-4423-b738-50476b66c833"). InnerVolumeSpecName "kube-api-access-bzrbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:42:57.003213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.003057 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:57.003213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.003092 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:57.003213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.003108 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df617b70-0871-4423-b738-50476b66c833-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:57.003213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.003123 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:57.003213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.003137 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:57.003213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.003153 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df617b70-0871-4423-b738-50476b66c833-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:57.003213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.003167 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df617b70-0871-4423-b738-50476b66c833-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:57.003213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.003181 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df617b70-0871-4423-b738-50476b66c833-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:57.003213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.003194 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzrbm\" (UniqueName: \"kubernetes.io/projected/df617b70-0871-4423-b738-50476b66c833-kube-api-access-bzrbm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:57.111675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.111639 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb"] Oct 31 19:42:57.124613 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.123814 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq"] Oct 31 19:42:57.142878 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.142805 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-dwnsq"] Oct 31 19:42:57.158940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.157342 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7" podStartSLOduration=10.157323191 podStartE2EDuration="10.157323191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:56.706041379 +0000 UTC m=+1411.068861753" watchObservedRunningTime="2026-10-31 19:42:57.157323191 +0000 UTC m=+1411.520143564" Oct 31 19:42:57.206785 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.206721 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk"] Oct 31 19:42:57.214835 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.214801 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-mg2hk"] Oct 31 19:42:57.570689 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.570642 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" event={"ID":"c86e67dd-5de9-4213-b26d-fcbc3bd90353","Type":"ContainerStarted","Data":"61e8372de968812a78948b9889ca8ea08d70b5f79321f7283cf56b354d18f00a"} Oct 31 19:42:57.570854 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.570713 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" event={"ID":"c86e67dd-5de9-4213-b26d-fcbc3bd90353","Type":"ContainerStarted","Data":"87f19639741372e2c282ea8c1536146ca6ea130f76cd25d5c2370ee5d5ea4160"} Oct 31 19:42:57.572542 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.572521 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" Oct 31 19:42:57.573566 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.572544 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p" event={"ID":"df617b70-0871-4423-b738-50476b66c833","Type":"ContainerDied","Data":"e65be1b7d0ff661aabd0e582be4eb6d4099aaaed5cd3a75d132dc83e51bf1a9a"} Oct 31 19:42:57.573566 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.573321 4910 scope.go:122] "RemoveContainer" containerID="508db727dd9f175dee66c2f1aa91cf0cbb554a09f6947c83a70ef71c9bca48e5" Oct 31 19:42:57.583066 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.583029 4910 generic.go:356] "Generic (PLEG): container finished" podID="7f403772-348e-44cb-adf1-c695cf83f1ee" containerID="8115cc157caf6bb6cebc5eb34257bac13cf78f45e1727245bd3ed3fa789de338" exitCode=0 Oct 31 19:42:57.583313 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.583150 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7" event={"ID":"7f403772-348e-44cb-adf1-c695cf83f1ee","Type":"ContainerDied","Data":"8115cc157caf6bb6cebc5eb34257bac13cf78f45e1727245bd3ed3fa789de338"} Oct 31 19:42:57.589563 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.589544 4910 scope.go:122] "RemoveContainer" containerID="0ee1fe21413c867adee167d9aac19f036cf7a8de9dc3f7c257ca2d48b0d929f5" Oct 31 19:42:57.597782 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.597727 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" podStartSLOduration=9.597711319 podStartE2EDuration="9.597711319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:57.595547738 +0000 UTC m=+1411.958368112" watchObservedRunningTime="2026-10-31 19:42:57.597711319 +0000 UTC m=+1411.960531691" Oct 31 19:42:57.623944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.623858 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p"] Oct 31 19:42:57.631634 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.631607 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-bbz5p"] Oct 31 19:42:57.710458 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.709228 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:57.870614 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.870569 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00f5bd2c-c4f2-4854-8169-dd65ad5d301d" path="/var/lib/kubelet/pods/00f5bd2c-c4f2-4854-8169-dd65ad5d301d/volumes" Oct 31 19:42:57.871418 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.871384 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="507b2fb7-a22a-4cac-9d4a-23199f7fcbff" path="/var/lib/kubelet/pods/507b2fb7-a22a-4cac-9d4a-23199f7fcbff/volumes" Oct 31 19:42:57.872399 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.872375 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df617b70-0871-4423-b738-50476b66c833" path="/var/lib/kubelet/pods/df617b70-0871-4423-b738-50476b66c833/volumes" Oct 31 19:42:57.873624 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:57.873577 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:58.598145 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:58.598119 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:58.599171 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:58.598886 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:42:58.920707 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:58.920677 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7" Oct 31 19:42:58.980069 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:58.980021 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f403772-348e-44cb-adf1-c695cf83f1ee-kube-api-access-2fv98\" (UniqueName: \"kubernetes.io/projected/7f403772-348e-44cb-adf1-c695cf83f1ee-kube-api-access-2fv98\") pod \"7f403772-348e-44cb-adf1-c695cf83f1ee\" (UID: \"7f403772-348e-44cb-adf1-c695cf83f1ee\") " Oct 31 19:42:58.980267 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:58.980096 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f403772-348e-44cb-adf1-c695cf83f1ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f403772-348e-44cb-adf1-c695cf83f1ee-maas-api-xks-ca\") pod \"7f403772-348e-44cb-adf1-c695cf83f1ee\" (UID: \"7f403772-348e-44cb-adf1-c695cf83f1ee\") " Oct 31 19:42:58.980466 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:58.980446 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f403772-348e-44cb-adf1-c695cf83f1ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f403772-348e-44cb-adf1-c695cf83f1ee-maas-api-openshift-ca\") pod \"7f403772-348e-44cb-adf1-c695cf83f1ee\" (UID: \"7f403772-348e-44cb-adf1-c695cf83f1ee\") " Oct 31 19:42:58.981533 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:58.981497 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f403772-348e-44cb-adf1-c695cf83f1ee-maas-api-openshift-ca" pod "7f403772-348e-44cb-adf1-c695cf83f1ee" (UID: "7f403772-348e-44cb-adf1-c695cf83f1ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:42:58.984694 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:58.984654 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f403772-348e-44cb-adf1-c695cf83f1ee-maas-api-xks-ca" pod "7f403772-348e-44cb-adf1-c695cf83f1ee" (UID: "7f403772-348e-44cb-adf1-c695cf83f1ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:42:58.985104 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:58.985069 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f403772-348e-44cb-adf1-c695cf83f1ee-kube-api-access-2fv98" pod "7f403772-348e-44cb-adf1-c695cf83f1ee" (UID: "7f403772-348e-44cb-adf1-c695cf83f1ee"). InnerVolumeSpecName "kube-api-access-2fv98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:42:59.084884 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:59.082392 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f403772-348e-44cb-adf1-c695cf83f1ee-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:59.084884 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:59.082420 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fv98\" (UniqueName: \"kubernetes.io/projected/7f403772-348e-44cb-adf1-c695cf83f1ee-kube-api-access-2fv98\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:59.084884 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:59.082431 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f403772-348e-44cb-adf1-c695cf83f1ee-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:42:59.610973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:59.610213 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7" Oct 31 19:42:59.610973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:59.610243 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7" event={"ID":"7f403772-348e-44cb-adf1-c695cf83f1ee","Type":"ContainerDied","Data":"701e920b48ece24cbf1890b02652142a6428627053316d968563aff4ed441ef5"} Oct 31 19:42:59.610973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:42:59.610310 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="701e920b48ece24cbf1890b02652142a6428627053316d968563aff4ed441ef5" Oct 31 19:43:00.887377 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:00.887340 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d319a-2t84q"] Oct 31 19:43:00.895530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:00.889840 4910 cpu_manager.go:414] "containerMap: removing container" podUID="df617b70-0871-4423-b738-50476b66c833" containerName="istio-proxy" Oct 31 19:43:00.895530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:00.889864 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df617b70-0871-4423-b738-50476b66c833" containerName="istio-proxy" Oct 31 19:43:00.895530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:00.889893 4910 cpu_manager.go:414] "containerMap: removing container" podUID="507b2fb7-a22a-4cac-9d4a-23199f7fcbff" containerName="istio-proxy" Oct 31 19:43:00.895530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:00.889901 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="507b2fb7-a22a-4cac-9d4a-23199f7fcbff" containerName="istio-proxy" Oct 31 19:43:00.895530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:00.889997 4910 cpu_manager.go:414] "containerMap: removing container" podUID="7f403772-348e-44cb-adf1-c695cf83f1ee" containerName="revoke-keys" Oct 31 19:43:00.895530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:00.890008 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f403772-348e-44cb-adf1-c695cf83f1ee" containerName="revoke-keys" Oct 31 19:43:00.895530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:00.890083 4910 cpu_manager.go:414] "containerMap: removing container" podUID="00f5bd2c-c4f2-4854-8169-dd65ad5d301d" containerName="istio-proxy" Oct 31 19:43:00.895530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:00.890091 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00f5bd2c-c4f2-4854-8169-dd65ad5d301d" containerName="istio-proxy" Oct 31 19:43:00.895530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:00.890593 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="507b2fb7-a22a-4cac-9d4a-23199f7fcbff" containerName="istio-proxy" Oct 31 19:43:00.895530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:00.890651 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f403772-348e-44cb-adf1-c695cf83f1ee" containerName="revoke-keys" Oct 31 19:43:00.895530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:00.890677 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="00f5bd2c-c4f2-4854-8169-dd65ad5d301d" containerName="istio-proxy" Oct 31 19:43:00.895530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:00.890699 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="df617b70-0871-4423-b738-50476b66c833" containerName="istio-proxy" Oct 31 19:43:00.912282 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:00.912227 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d319a-2t84q" Oct 31 19:43:00.917525 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:00.917490 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d319a-2t84q"] Oct 31 19:43:00.919410 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:00.918065 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:43:00.923141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:00.923116 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:43:01.020543 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:01.020503 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrz69\" (UniqueName: \"kubernetes.io/projected/50252ae6-f1ab-4a90-a33d-36b0ed477a49-kube-api-access-mrz69\") pod \"maas-api-revoke-keys-e2e-api-a-2d319a-2t84q\" (UID: \"50252ae6-f1ab-4a90-a33d-36b0ed477a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d319a-2t84q" Oct 31 19:43:01.020928 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:01.020695 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50252ae6-f1ab-4a90-a33d-36b0ed477a49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2d319a-2t84q\" (UID: \"50252ae6-f1ab-4a90-a33d-36b0ed477a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d319a-2t84q" Oct 31 19:43:01.020928 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:01.020772 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50252ae6-f1ab-4a90-a33d-36b0ed477a49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2d319a-2t84q\" (UID: \"50252ae6-f1ab-4a90-a33d-36b0ed477a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d319a-2t84q" Oct 31 19:43:01.122768 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:01.122728 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrz69\" (UniqueName: \"kubernetes.io/projected/50252ae6-f1ab-4a90-a33d-36b0ed477a49-kube-api-access-mrz69\") pod \"maas-api-revoke-keys-e2e-api-a-2d319a-2t84q\" (UID: \"50252ae6-f1ab-4a90-a33d-36b0ed477a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d319a-2t84q" Oct 31 19:43:01.123060 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:01.122947 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50252ae6-f1ab-4a90-a33d-36b0ed477a49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2d319a-2t84q\" (UID: \"50252ae6-f1ab-4a90-a33d-36b0ed477a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d319a-2t84q" Oct 31 19:43:01.123060 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:01.123030 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50252ae6-f1ab-4a90-a33d-36b0ed477a49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2d319a-2t84q\" (UID: \"50252ae6-f1ab-4a90-a33d-36b0ed477a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d319a-2t84q" Oct 31 19:43:01.124092 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:01.124057 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50252ae6-f1ab-4a90-a33d-36b0ed477a49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2d319a-2t84q\" (UID: \"50252ae6-f1ab-4a90-a33d-36b0ed477a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d319a-2t84q" Oct 31 19:43:01.127323 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:01.127283 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50252ae6-f1ab-4a90-a33d-36b0ed477a49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2d319a-2t84q\" (UID: \"50252ae6-f1ab-4a90-a33d-36b0ed477a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d319a-2t84q" Oct 31 19:43:01.139650 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:01.139578 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrz69\" (UniqueName: \"kubernetes.io/projected/50252ae6-f1ab-4a90-a33d-36b0ed477a49-kube-api-access-mrz69\") pod \"maas-api-revoke-keys-e2e-api-a-2d319a-2t84q\" (UID: \"50252ae6-f1ab-4a90-a33d-36b0ed477a49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d319a-2t84q" Oct 31 19:43:01.270407 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:01.270371 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d319a-2t84q" Oct 31 19:43:01.332202 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:01.332156 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb"] Oct 31 19:43:01.333420 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:01.333395 4910 cpu_manager.go:414] "containerMap: removing container" podUID="507b2fb7-a22a-4cac-9d4a-23199f7fcbff" containerName="istio-proxy" Oct 31 19:43:01.333524 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:01.333424 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="507b2fb7-a22a-4cac-9d4a-23199f7fcbff" containerName="istio-proxy" Oct 31 19:43:01.333646 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:01.333626 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="507b2fb7-a22a-4cac-9d4a-23199f7fcbff" containerName="istio-proxy" Oct 31 19:43:01.345730 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:01.345702 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb"] Oct 31 19:43:01.346085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:01.346065 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb" Oct 31 19:43:01.354525 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:01.354365 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1e8cf0\"" Oct 31 19:43:01.354525 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:01.354497 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1e8cf0-dockercfg-8l2cj\"" Oct 31 19:43:01.446092 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:01.436005 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62d760d9-672c-4690-b5de-e8c6456c5431-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb\" (UID: \"62d760d9-672c-4690-b5de-e8c6456c5431\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb" Oct 31 19:43:01.446725 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:01.446454 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gnkp\" (UniqueName: \"kubernetes.io/projected/62d760d9-672c-4690-b5de-e8c6456c5431-kube-api-access-5gnkp\") pod \"payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb\" (UID: \"62d760d9-672c-4690-b5de-e8c6456c5431\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb" Oct 31 19:43:01.568363 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:01.565845 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62d760d9-672c-4690-b5de-e8c6456c5431-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb\" (UID: \"62d760d9-672c-4690-b5de-e8c6456c5431\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb" Oct 31 19:43:01.568363 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:01.565944 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gnkp\" (UniqueName: \"kubernetes.io/projected/62d760d9-672c-4690-b5de-e8c6456c5431-kube-api-access-5gnkp\") pod \"payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb\" (UID: \"62d760d9-672c-4690-b5de-e8c6456c5431\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb" Oct 31 19:43:01.568363 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:01.567144 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62d760d9-672c-4690-b5de-e8c6456c5431-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb\" (UID: \"62d760d9-672c-4690-b5de-e8c6456c5431\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb" Oct 31 19:43:01.683482 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:01.683441 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gnkp\" (UniqueName: \"kubernetes.io/projected/62d760d9-672c-4690-b5de-e8c6456c5431-kube-api-access-5gnkp\") pod \"payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb\" (UID: \"62d760d9-672c-4690-b5de-e8c6456c5431\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb" Oct 31 19:43:01.683762 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:01.683744 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb" Oct 31 19:43:02.004558 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.004526 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d319a-2t84q"] Oct 31 19:43:02.005039 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:43:02.004748 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50252ae6_f1ab_4a90_a33d_36b0ed477a49.slice/crio-bcfab84c078b429d19bdb15e3073ddf12bc2b618c8f6cacf74ad7173b19cad6c WatchSource:0}: Error finding container bcfab84c078b429d19bdb15e3073ddf12bc2b618c8f6cacf74ad7173b19cad6c: Status 404 returned error can't find the container with id bcfab84c078b429d19bdb15e3073ddf12bc2b618c8f6cacf74ad7173b19cad6c Oct 31 19:43:02.133355 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.133323 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb"] Oct 31 19:43:02.148299 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.148261 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch"] Oct 31 19:43:02.156620 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.156579 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.178555 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.178515 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch"] Oct 31 19:43:02.307621 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.307539 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rbb6\" (UniqueName: \"kubernetes.io/projected/199b78b1-0dcc-44df-a25b-9c401a86ad70-kube-api-access-6rbb6\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.307835 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.307677 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-token\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.307835 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.307769 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/199b78b1-0dcc-44df-a25b-9c401a86ad70-istiod-ca-cert\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.308732 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.307851 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-workload-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.308732 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.307882 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-data\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.308732 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.308165 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-workload-certs\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.308732 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.308233 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-envoy\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.308732 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.308283 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-credential-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.308732 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.308400 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-podinfo\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.416470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.410145 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-workload-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.416470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.409697 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-workload-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.416470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.410480 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-data\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.416470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.410626 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-workload-certs\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.416470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.410666 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-envoy\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.416470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.410695 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-credential-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.416470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.410752 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-podinfo\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.416470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.410776 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rbb6\" (UniqueName: \"kubernetes.io/projected/199b78b1-0dcc-44df-a25b-9c401a86ad70-kube-api-access-6rbb6\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.416470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.410825 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-token\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.416470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.410890 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/199b78b1-0dcc-44df-a25b-9c401a86ad70-istiod-ca-cert\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.416470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.411331 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-credential-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.416470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.411544 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-workload-certs\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.416470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.411571 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-data\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.416470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.411941 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/199b78b1-0dcc-44df-a25b-9c401a86ad70-istiod-ca-cert\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.416470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.414331 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-podinfo\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.416470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.414569 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-envoy\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.427013 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.426986 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rbb6\" (UniqueName: \"kubernetes.io/projected/199b78b1-0dcc-44df-a25b-9c401a86ad70-kube-api-access-6rbb6\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.435307 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.435282 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-token\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.665772 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.665667 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:02.724585 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.724543 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb" event={"ID":"62d760d9-672c-4690-b5de-e8c6456c5431","Type":"ContainerStarted","Data":"1f8427e1acc867c2d79b638d77272c277c74ca79b003de08f87083d6aff05c9e"} Oct 31 19:43:02.737624 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.724661 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb" event={"ID":"62d760d9-672c-4690-b5de-e8c6456c5431","Type":"ContainerStarted","Data":"9ac3dec9a187f8558b06685c2ac9ac9874ba45cba0316954d70af001bdebcfa4"} Oct 31 19:43:02.737624 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.725769 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb" Oct 31 19:43:02.737624 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.728295 4910 generic.go:356] "Generic (PLEG): container finished" podID="4ca75657-04c9-4283-9059-220e69657f7c" containerID="19c829415c183e4e9f2ad1ef1a25777173810df061bbd00f5d627fb439bf6d37" exitCode=0 Oct 31 19:43:02.737624 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.728422 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" event={"ID":"4ca75657-04c9-4283-9059-220e69657f7c","Type":"ContainerDied","Data":"19c829415c183e4e9f2ad1ef1a25777173810df061bbd00f5d627fb439bf6d37"} Oct 31 19:43:02.737624 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.729039 4910 scope.go:122] "RemoveContainer" containerID="19c829415c183e4e9f2ad1ef1a25777173810df061bbd00f5d627fb439bf6d37" Oct 31 19:43:02.737624 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.736772 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:43:02.737624 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.736865 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:43:02.738604 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.738243 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:43:02.746825 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.746653 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d319a-2t84q" event={"ID":"50252ae6-f1ab-4a90-a33d-36b0ed477a49","Type":"ContainerStarted","Data":"5cbfeabcd5fd57973c570c07874a694e25b026d82397df92014d39152de8dd67"} Oct 31 19:43:02.747993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.747003 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d319a-2t84q" event={"ID":"50252ae6-f1ab-4a90-a33d-36b0ed477a49","Type":"ContainerStarted","Data":"bcfab84c078b429d19bdb15e3073ddf12bc2b618c8f6cacf74ad7173b19cad6c"} Oct 31 19:43:02.769447 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.769251 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb" podStartSLOduration=1.76923581 podStartE2EDuration="1.76923581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:02.767841959 +0000 UTC m=+1417.130662334" watchObservedRunningTime="2026-10-31 19:43:02.76923581 +0000 UTC m=+1417.132056181" Oct 31 19:43:02.790706 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.790648 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d319a-2t84q" podStartSLOduration=2.790635357 podStartE2EDuration="2.790635357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:02.786609444 +0000 UTC m=+1417.149429817" watchObservedRunningTime="2026-10-31 19:43:02.790635357 +0000 UTC m=+1417.153455729" Oct 31 19:43:02.971823 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.971796 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch"] Oct 31 19:43:02.975202 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:43:02.975165 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199b78b1_0dcc_44df_a25b_9c401a86ad70.slice/crio-4b9cb99e587114ee41326bc96fb4fa6af84759be806b042114a65c0e58308e45 WatchSource:0}: Error finding container 4b9cb99e587114ee41326bc96fb4fa6af84759be806b042114a65c0e58308e45: Status 404 returned error can't find the container with id 4b9cb99e587114ee41326bc96fb4fa6af84759be806b042114a65c0e58308e45 Oct 31 19:43:02.979358 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.979314 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:43:02.979440 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.979410 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:43:02.979486 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:02.979453 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:43:03.775115 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:03.775069 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" event={"ID":"4ca75657-04c9-4283-9059-220e69657f7c","Type":"ContainerStarted","Data":"4a3cea06a4855de38ba869100c1ccd037ca61a071422c740548fcc7204a8f628"} Oct 31 19:43:03.778507 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:03.778472 4910 generic.go:356] "Generic (PLEG): container finished" podID="50252ae6-f1ab-4a90-a33d-36b0ed477a49" containerID="5cbfeabcd5fd57973c570c07874a694e25b026d82397df92014d39152de8dd67" exitCode=0 Oct 31 19:43:03.778748 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:03.778704 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d319a-2t84q" event={"ID":"50252ae6-f1ab-4a90-a33d-36b0ed477a49","Type":"ContainerDied","Data":"5cbfeabcd5fd57973c570c07874a694e25b026d82397df92014d39152de8dd67"} Oct 31 19:43:03.782393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:03.782364 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" event={"ID":"199b78b1-0dcc-44df-a25b-9c401a86ad70","Type":"ContainerStarted","Data":"64490afe30f522a499652851d571209250fcba5b54658150b7da616bfad14564"} Oct 31 19:43:03.782639 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:03.782606 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" event={"ID":"199b78b1-0dcc-44df-a25b-9c401a86ad70","Type":"ContainerStarted","Data":"4b9cb99e587114ee41326bc96fb4fa6af84759be806b042114a65c0e58308e45"} Oct 31 19:43:03.785900 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:03.785871 4910 generic.go:356] "Generic (PLEG): container finished" podID="499b41f2-15df-4636-abbc-35fa6f8a164a" containerID="ee5138499d94a28ba9e5fff1962310587ee29c357947bdfd6e3be726302de64c" exitCode=0 Oct 31 19:43:03.787010 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:03.786090 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" event={"ID":"499b41f2-15df-4636-abbc-35fa6f8a164a","Type":"ContainerDied","Data":"ee5138499d94a28ba9e5fff1962310587ee29c357947bdfd6e3be726302de64c"} Oct 31 19:43:03.787644 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:03.787627 4910 scope.go:122] "RemoveContainer" containerID="ee5138499d94a28ba9e5fff1962310587ee29c357947bdfd6e3be726302de64c" Oct 31 19:43:03.790502 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:03.790431 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:43:03.790693 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:03.790672 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:43:03.790829 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:03.790814 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:43:03.808011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:03.807528 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:43:03.808011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:03.807665 4910 patch_prober.go:29] interesting pod/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 19:43:03.808011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:03.807713 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" podUID="4ca75657-04c9-4283-9059-220e69657f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 19:43:03.846621 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:03.846519 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" podStartSLOduration=1.846498495 podStartE2EDuration="1.846498495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:03.827397031 +0000 UTC m=+1418.190217404" watchObservedRunningTime="2026-10-31 19:43:03.846498495 +0000 UTC m=+1418.209318868" Oct 31 19:43:04.683981 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:04.683940 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:04.684182 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:04.684083 4910 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 31 19:43:04.684182 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:04.684123 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" podUID="199b78b1-0dcc-44df-a25b-9c401a86ad70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 31 19:43:04.805502 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:04.805466 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" event={"ID":"499b41f2-15df-4636-abbc-35fa6f8a164a","Type":"ContainerStarted","Data":"853d31bcc99e84aa887bf15f15988fde746f71055e45e99b9091cf9f3f22322f"} Oct 31 19:43:04.806710 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:04.806689 4910 patch_prober.go:29] interesting pod/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 19:43:04.806789 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:04.806725 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" podUID="4ca75657-04c9-4283-9059-220e69657f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 19:43:05.595211 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:05.594855 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:43:05.639936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:05.602109 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:43:05.639936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:05.625210 4910 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 31 19:43:05.639936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:05.625458 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" podUID="499b41f2-15df-4636-abbc-35fa6f8a164a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 31 19:43:05.701935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:05.669330 4910 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 31 19:43:05.702267 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:05.702222 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" podUID="199b78b1-0dcc-44df-a25b-9c401a86ad70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 31 19:43:05.809198 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:05.809148 4910 patch_prober.go:29] interesting pod/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 19:43:05.809806 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:05.809770 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" podUID="4ca75657-04c9-4283-9059-220e69657f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 19:43:06.078867 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:06.070972 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:06.198439 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:06.198406 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d319a-2t84q" Oct 31 19:43:06.248092 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:06.248048 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:06.249323 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:06.249294 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:06.384990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:06.352061 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50252ae6-f1ab-4a90-a33d-36b0ed477a49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50252ae6-f1ab-4a90-a33d-36b0ed477a49-maas-api-xks-ca\") pod \"50252ae6-f1ab-4a90-a33d-36b0ed477a49\" (UID: \"50252ae6-f1ab-4a90-a33d-36b0ed477a49\") " Oct 31 19:43:06.384990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:06.352145 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50252ae6-f1ab-4a90-a33d-36b0ed477a49-kube-api-access-mrz69\" (UniqueName: \"kubernetes.io/projected/50252ae6-f1ab-4a90-a33d-36b0ed477a49-kube-api-access-mrz69\") pod \"50252ae6-f1ab-4a90-a33d-36b0ed477a49\" (UID: \"50252ae6-f1ab-4a90-a33d-36b0ed477a49\") " Oct 31 19:43:06.384990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:06.352349 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50252ae6-f1ab-4a90-a33d-36b0ed477a49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50252ae6-f1ab-4a90-a33d-36b0ed477a49-maas-api-openshift-ca\") pod \"50252ae6-f1ab-4a90-a33d-36b0ed477a49\" (UID: \"50252ae6-f1ab-4a90-a33d-36b0ed477a49\") " Oct 31 19:43:06.451984 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:06.451603 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50252ae6-f1ab-4a90-a33d-36b0ed477a49-maas-api-openshift-ca" pod "50252ae6-f1ab-4a90-a33d-36b0ed477a49" (UID: "50252ae6-f1ab-4a90-a33d-36b0ed477a49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:43:06.503937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:06.503642 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50252ae6-f1ab-4a90-a33d-36b0ed477a49-maas-api-xks-ca" pod "50252ae6-f1ab-4a90-a33d-36b0ed477a49" (UID: "50252ae6-f1ab-4a90-a33d-36b0ed477a49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:43:06.513224 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:06.511617 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50252ae6-f1ab-4a90-a33d-36b0ed477a49-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:43:06.513224 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:06.511655 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50252ae6-f1ab-4a90-a33d-36b0ed477a49-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:43:06.540505 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:06.523278 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50252ae6-f1ab-4a90-a33d-36b0ed477a49-kube-api-access-mrz69" pod "50252ae6-f1ab-4a90-a33d-36b0ed477a49" (UID: "50252ae6-f1ab-4a90-a33d-36b0ed477a49"). InnerVolumeSpecName "kube-api-access-mrz69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:43:06.569610 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:06.569536 4910 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 31 19:43:06.569920 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:06.569847 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" podUID="499b41f2-15df-4636-abbc-35fa6f8a164a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 31 19:43:06.570026 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:06.569988 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:43:06.595044 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:06.594998 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:06.672044 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:06.671945 4910 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 31 19:43:06.672310 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:06.672268 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" podUID="199b78b1-0dcc-44df-a25b-9c401a86ad70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 31 19:43:06.710466 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:06.672429 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrz69\" (UniqueName: \"kubernetes.io/projected/50252ae6-f1ab-4a90-a33d-36b0ed477a49-kube-api-access-mrz69\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:43:06.808982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:06.808944 4910 patch_prober.go:29] interesting pod/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 19:43:06.809260 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:06.809228 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" podUID="4ca75657-04c9-4283-9059-220e69657f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 19:43:07.130620 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.130570 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:43:07.157951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.150438 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:07.410072 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.153210 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:43:07.410072 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.153256 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 19:43:07.410072 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.178066 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:07.410072 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.178144 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:07.410072 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.242929 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:07.410072 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.289036 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:07.410072 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.289075 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:07.410072 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.315853 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b"] Oct 31 19:43:07.486936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.454876 4910 cpu_manager.go:414] "containerMap: removing container" podUID="50252ae6-f1ab-4a90-a33d-36b0ed477a49" containerName="revoke-keys" Oct 31 19:43:07.487198 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.487172 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50252ae6-f1ab-4a90-a33d-36b0ed477a49" containerName="revoke-keys" Oct 31 19:43:07.503975 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.496764 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="50252ae6-f1ab-4a90-a33d-36b0ed477a49" containerName="revoke-keys" Oct 31 19:43:07.550473 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.550439 4910 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 31 19:43:07.550763 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.550719 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" podUID="499b41f2-15df-4636-abbc-35fa6f8a164a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 31 19:43:07.589147 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.589090 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:07.594053 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.589841 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:43:07.632070 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.604085 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d319a-2t84q" Oct 31 19:43:07.668202 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.668110 4910 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 31 19:43:07.668540 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.668482 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" podUID="199b78b1-0dcc-44df-a25b-9c401a86ad70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 31 19:43:07.668803 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.668777 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:07.926669 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.926396 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b" Oct 31 19:43:07.927277 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.927240 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:07.942467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.941742 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d45f6c-serving-cert\"" Oct 31 19:43:07.942467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.941818 4910 patch_prober.go:29] interesting pod/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 19:43:07.942467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:07.941856 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" podUID="4ca75657-04c9-4283-9059-220e69657f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 19:43:08.086933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.084311 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b"] Oct 31 19:43:08.086933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.084494 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:43:08.086933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.084518 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d319a-2t84q" event={"ID":"50252ae6-f1ab-4a90-a33d-36b0ed477a49","Type":"ContainerDied","Data":"bcfab84c078b429d19bdb15e3073ddf12bc2b618c8f6cacf74ad7173b19cad6c"} Oct 31 19:43:08.086933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.084542 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcfab84c078b429d19bdb15e3073ddf12bc2b618c8f6cacf74ad7173b19cad6c" Oct 31 19:43:08.086933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.084578 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:43:08.086933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.084618 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:43:08.086933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.084651 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:43:08.086933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.084690 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:43:08.111935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.099313 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:08.138978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.138935 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:08.229940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.204395 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpvvf\" (UniqueName: \"kubernetes.io/projected/69d2da79-5a06-4c5a-9185-58c9136a05cf-kube-api-access-jpvvf\") pod \"maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b\" (UID: \"69d2da79-5a06-4c5a-9185-58c9136a05cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b" Oct 31 19:43:08.303567 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.300959 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69d2da79-5a06-4c5a-9185-58c9136a05cf-metrics-certs\") pod \"maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b\" (UID: \"69d2da79-5a06-4c5a-9185-58c9136a05cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b" Oct 31 19:43:08.303567 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.301247 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69d2da79-5a06-4c5a-9185-58c9136a05cf-maas-api-tls\") pod \"maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b\" (UID: \"69d2da79-5a06-4c5a-9185-58c9136a05cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b" Oct 31 19:43:08.303567 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.206722 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:43:08.303567 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.301689 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d2da79-5a06-4c5a-9185-58c9136a05cf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b\" (UID: \"69d2da79-5a06-4c5a-9185-58c9136a05cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b" Oct 31 19:43:08.362590 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.362559 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:43:08.499003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.461665 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpvvf\" (UniqueName: \"kubernetes.io/projected/69d2da79-5a06-4c5a-9185-58c9136a05cf-kube-api-access-jpvvf\") pod \"maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b\" (UID: \"69d2da79-5a06-4c5a-9185-58c9136a05cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b" Oct 31 19:43:08.513851 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.506231 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69d2da79-5a06-4c5a-9185-58c9136a05cf-metrics-certs\") pod \"maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b\" (UID: \"69d2da79-5a06-4c5a-9185-58c9136a05cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b" Oct 31 19:43:08.541043 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.541007 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69d2da79-5a06-4c5a-9185-58c9136a05cf-maas-api-tls\") pod \"maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b\" (UID: \"69d2da79-5a06-4c5a-9185-58c9136a05cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b" Oct 31 19:43:08.541349 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.541326 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d2da79-5a06-4c5a-9185-58c9136a05cf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b\" (UID: \"69d2da79-5a06-4c5a-9185-58c9136a05cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b" Oct 31 19:43:08.566938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.525004 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": context deadline exceeded" Oct 31 19:43:08.863868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.537735 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:08.885941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.537765 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:08.896028 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.537789 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:43:08.896552 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.537820 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:43:08.910868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.540129 4910 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 31 19:43:08.911120 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.911092 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" podUID="499b41f2-15df-4636-abbc-35fa6f8a164a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 31 19:43:08.911226 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.537695 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:43:08.926973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.537656 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:08.972188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.586543 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq"] Oct 31 19:43:08.972188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.944567 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:43:08.972188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.944639 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:43:08.972188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.944675 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:43:08.972188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.616751 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:43:08.972188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.817408 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpvvf\" (UniqueName: \"kubernetes.io/projected/69d2da79-5a06-4c5a-9185-58c9136a05cf-kube-api-access-jpvvf\") pod \"maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b\" (UID: \"69d2da79-5a06-4c5a-9185-58c9136a05cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b" Oct 31 19:43:08.972188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.545664 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d2da79-5a06-4c5a-9185-58c9136a05cf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b\" (UID: \"69d2da79-5a06-4c5a-9185-58c9136a05cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b" Oct 31 19:43:08.972188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.883986 4910 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 31 19:43:08.972188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.947075 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" podUID="199b78b1-0dcc-44df-a25b-9c401a86ad70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 31 19:43:08.972188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.897506 4910 patch_prober.go:29] interesting pod/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 19:43:08.972188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.947126 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" podUID="4ca75657-04c9-4283-9059-220e69657f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 19:43:08.972188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.616786 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:43:08.972188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.709611 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69d2da79-5a06-4c5a-9185-58c9136a05cf-maas-api-tls\") pod \"maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b\" (UID: \"69d2da79-5a06-4c5a-9185-58c9136a05cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b" Oct 31 19:43:08.972188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.967649 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:43:08.975832 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.850512 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded" Oct 31 19:43:08.979930 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.976370 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:43:08.988166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:08.988139 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:43:09.066875 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:09.066663 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:09.157498 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:09.099711 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:43:09.157498 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:09.156087 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:43:09.157498 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:09.101664 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69d2da79-5a06-4c5a-9185-58c9136a05cf-metrics-certs\") pod \"maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b\" (UID: \"69d2da79-5a06-4c5a-9185-58c9136a05cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b" Oct 31 19:43:09.179161 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:09.164873 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:09.257988 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:09.257180 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:43:09.274814 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:09.274773 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:09.369953 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:09.368771 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b" Oct 31 19:43:09.664934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:09.648732 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:43:09.738092 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:09.668767 4910 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 31 19:43:09.763652 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:09.763607 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" podUID="199b78b1-0dcc-44df-a25b-9c401a86ad70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 31 19:43:09.869001 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:09.696853 4910 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 31 19:43:09.887406 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:09.887357 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" podUID="499b41f2-15df-4636-abbc-35fa6f8a164a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 31 19:43:09.887707 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:09.869215 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:09.953972 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:09.945862 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:43:09.953972 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:09.946346 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:09.991312 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:09.989991 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:10.015303 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:09.991593 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:10.185982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:10.182030 4910 patch_prober.go:29] interesting pod/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 19:43:10.185982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:10.182101 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" podUID="4ca75657-04c9-4283-9059-220e69657f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 19:43:10.221467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:10.221422 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 19:43:10.369092 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:10.226495 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:43:10.513324 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:10.513217 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:10.514725 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:10.514677 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:10.609709 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:10.563245 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:10.634817 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:10.634782 4910 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 31 19:43:10.635134 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:10.635086 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" podUID="499b41f2-15df-4636-abbc-35fa6f8a164a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 31 19:43:10.674039 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:10.661419 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:10.709065 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:10.709029 4910 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Oct 31 19:43:10.747934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:10.710959 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" podUID="199b78b1-0dcc-44df-a25b-9c401a86ad70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Oct 31 19:43:10.915089 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:10.708276 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:10.915089 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:10.731006 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:43:10.915089 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:10.751701 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:43:10.915089 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:10.906950 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:43:10.915089 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:10.893239 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:43:10.915089 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:10.846340 4910 patch_prober.go:29] interesting pod/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 31 19:43:10.915089 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:10.909030 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" podUID="4ca75657-04c9-4283-9059-220e69657f7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 31 19:43:10.981935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:10.943847 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:11.011206 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:10.963672 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:11.012223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:10.963537 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:11.040932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:11.016035 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:43:11.040932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:11.017779 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:43:11.091939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:11.030719 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:11.295584 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:11.292837 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:43:11.295584 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:11.294904 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:43:11.295584 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:11.294974 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:43:11.312380 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:11.312352 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:43:11.372814 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:11.372767 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:11.381230 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:11.380928 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:43:11.461999 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:11.461790 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:43:11.548364 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:11.510141 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:43:11.548364 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:11.521163 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:43:11.578992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:11.562933 4910 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 31 19:43:11.578992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:11.562999 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" podUID="499b41f2-15df-4636-abbc-35fa6f8a164a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 31 19:43:11.589159 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:11.589115 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:43:12.210349 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:43:12.200874 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d2da79_5a06_4c5a_9185_58c9136a05cf.slice/crio-8484f97bb9fc9201ffaae0d40a1c59edff9947e84de18d881b2a80a786528f6d WatchSource:0}: Error finding container 8484f97bb9fc9201ffaae0d40a1c59edff9947e84de18d881b2a80a786528f6d: Status 404 returned error can't find the container with id 8484f97bb9fc9201ffaae0d40a1c59edff9947e84de18d881b2a80a786528f6d Oct 31 19:43:12.279760 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:12.279722 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:12.280078 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:12.280059 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b"] Oct 31 19:43:12.307752 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:12.307725 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:12.314956 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:12.312057 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:43:12.533616 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:12.533354 4910 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 31 19:43:12.533616 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:12.533414 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" podUID="499b41f2-15df-4636-abbc-35fa6f8a164a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 31 19:43:12.578547 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:12.578423 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:43:12.582119 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:12.580137 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:43:12.729537 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:43:12.729499 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:43:13.229586 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:13.229544 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b" event={"ID":"69d2da79-5a06-4c5a-9185-58c9136a05cf","Type":"ContainerStarted","Data":"c1f10e919922d3effa2bbec320f65da467f106ffe1d2bf7a35e8a7154295d3cb"} Oct 31 19:43:13.230091 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:13.229615 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b" event={"ID":"69d2da79-5a06-4c5a-9185-58c9136a05cf","Type":"ContainerStarted","Data":"8484f97bb9fc9201ffaae0d40a1c59edff9947e84de18d881b2a80a786528f6d"} Oct 31 19:43:13.251865 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:13.251816 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b" podStartSLOduration=6.787589974 podStartE2EDuration="7.25180217s" totalImagesPullingTime="464.212196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:13.251295392 +0000 UTC m=+1427.614115765" watchObservedRunningTime="2026-10-31 19:43:13.25180217 +0000 UTC m=+1427.614622541" Oct 31 19:43:13.537975 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:13.537877 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:43:13.541534 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:13.541502 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:43:13.791194 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:13.791110 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb" Oct 31 19:43:13.947764 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:13.947669 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq" podUID="cc611929-0e53-4f9f-88f7-89beb3685be8" containerName="payload-processing" containerID="cri-o://737394ed8be0a0df338a13655f4c57fc175ef90237b15e2147ff9eb01b136221" gracePeriod=25 Oct 31 19:43:14.275894 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:14.275862 4910 generic.go:356] "Generic (PLEG): container finished" podID="cc611929-0e53-4f9f-88f7-89beb3685be8" containerID="737394ed8be0a0df338a13655f4c57fc175ef90237b15e2147ff9eb01b136221" exitCode=0 Oct 31 19:43:14.276353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:14.275946 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq" event={"ID":"cc611929-0e53-4f9f-88f7-89beb3685be8","Type":"ContainerDied","Data":"737394ed8be0a0df338a13655f4c57fc175ef90237b15e2147ff9eb01b136221"} Oct 31 19:43:14.276555 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:14.276530 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b" Oct 31 19:43:14.309139 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:14.309110 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq" Oct 31 19:43:14.454794 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:14.454708 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc611929-0e53-4f9f-88f7-89beb3685be8-kube-api-access-b8878\" (UniqueName: \"kubernetes.io/projected/cc611929-0e53-4f9f-88f7-89beb3685be8-kube-api-access-b8878\") pod \"cc611929-0e53-4f9f-88f7-89beb3685be8\" (UID: \"cc611929-0e53-4f9f-88f7-89beb3685be8\") " Oct 31 19:43:14.454975 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:14.454817 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc611929-0e53-4f9f-88f7-89beb3685be8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc611929-0e53-4f9f-88f7-89beb3685be8-plugins-config-volume\") pod \"cc611929-0e53-4f9f-88f7-89beb3685be8\" (UID: \"cc611929-0e53-4f9f-88f7-89beb3685be8\") " Oct 31 19:43:14.455356 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:14.455331 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc611929-0e53-4f9f-88f7-89beb3685be8-plugins-config-volume" pod "cc611929-0e53-4f9f-88f7-89beb3685be8" (UID: "cc611929-0e53-4f9f-88f7-89beb3685be8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:43:14.455577 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:14.455554 4910 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc611929-0e53-4f9f-88f7-89beb3685be8-plugins-config-volume\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:43:14.457145 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:14.457119 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc611929-0e53-4f9f-88f7-89beb3685be8-kube-api-access-b8878" pod "cc611929-0e53-4f9f-88f7-89beb3685be8" (UID: "cc611929-0e53-4f9f-88f7-89beb3685be8"). InnerVolumeSpecName "kube-api-access-b8878". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:43:14.557051 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:14.557021 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8878\" (UniqueName: \"kubernetes.io/projected/cc611929-0e53-4f9f-88f7-89beb3685be8-kube-api-access-b8878\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:43:15.306968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:15.303196 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq" Oct 31 19:43:15.306968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:15.303220 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq" event={"ID":"cc611929-0e53-4f9f-88f7-89beb3685be8","Type":"ContainerDied","Data":"bfefe43595f1a172d285a8790fe9887c5b172bef57d6bb7ef6c76367e4ed166f"} Oct 31 19:43:15.306968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:15.303336 4910 scope.go:122] "RemoveContainer" containerID="737394ed8be0a0df338a13655f4c57fc175ef90237b15e2147ff9eb01b136221" Oct 31 19:43:15.334358 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:15.334319 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq"] Oct 31 19:43:15.339077 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:15.339049 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1c0c1d-758fdd696f-ff9zq"] Oct 31 19:43:15.761167 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:15.761091 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc611929-0e53-4f9f-88f7-89beb3685be8" path="/var/lib/kubelet/pods/cc611929-0e53-4f9f-88f7-89beb3685be8/volumes" Oct 31 19:43:20.310066 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:20.310031 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b" Oct 31 19:43:25.081635 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:25.081587 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8"] Oct 31 19:43:25.082782 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:25.082755 4910 cpu_manager.go:414] "containerMap: removing container" podUID="cc611929-0e53-4f9f-88f7-89beb3685be8" containerName="payload-processing" Oct 31 19:43:25.082782 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:25.082783 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc611929-0e53-4f9f-88f7-89beb3685be8" containerName="payload-processing" Oct 31 19:43:25.083034 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:25.083012 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc611929-0e53-4f9f-88f7-89beb3685be8" containerName="payload-processing" Oct 31 19:43:25.089146 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:25.089121 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8" Oct 31 19:43:25.105390 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:25.101873 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1e8cf0-serving-cert\"" Oct 31 19:43:25.105390 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:25.105175 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8"] Oct 31 19:43:25.205108 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:25.205065 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8\" (UID: \"d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8" Oct 31 19:43:25.205332 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:25.205119 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-maas-api-tls\") pod \"maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8\" (UID: \"d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8" Oct 31 19:43:25.205332 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:25.205315 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bssdn\" (UniqueName: \"kubernetes.io/projected/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-kube-api-access-bssdn\") pod \"maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8\" (UID: \"d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8" Oct 31 19:43:25.205432 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:25.205421 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-metrics-certs\") pod \"maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8\" (UID: \"d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8" Oct 31 19:43:25.310841 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:25.310294 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8\" (UID: \"d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8" Oct 31 19:43:25.310841 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:25.310362 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-maas-api-tls\") pod \"maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8\" (UID: \"d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8" Oct 31 19:43:25.310841 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:25.310582 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bssdn\" (UniqueName: \"kubernetes.io/projected/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-kube-api-access-bssdn\") pod \"maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8\" (UID: \"d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8" Oct 31 19:43:25.310841 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:25.310682 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-metrics-certs\") pod \"maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8\" (UID: \"d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8" Oct 31 19:43:25.322892 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:25.322859 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-metrics-certs\") pod \"maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8\" (UID: \"d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8" Oct 31 19:43:25.323472 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:25.323446 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8\" (UID: \"d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8" Oct 31 19:43:25.351784 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:25.351669 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-maas-api-tls\") pod \"maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8\" (UID: \"d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8" Oct 31 19:43:25.353771 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:25.353742 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bssdn\" (UniqueName: \"kubernetes.io/projected/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-kube-api-access-bssdn\") pod \"maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8\" (UID: \"d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8" Oct 31 19:43:25.417411 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:25.417377 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8" Oct 31 19:43:26.014748 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:26.014658 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8"] Oct 31 19:43:26.626999 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:26.626853 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8" event={"ID":"d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f","Type":"ContainerStarted","Data":"fa781564f45b28cf918599d797185df7ebb2ec36fe6d067106b56d054ba19e07"} Oct 31 19:43:26.626999 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:26.626977 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8" Oct 31 19:43:26.627538 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:26.627004 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8" event={"ID":"d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f","Type":"ContainerStarted","Data":"3abb0e50e1a06e0372eb8f7af40ef6fc2d459385aeda13617584000a93913a5b"} Oct 31 19:43:26.652136 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:26.652065 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8" podStartSLOduration=1.055803596 podStartE2EDuration="1.652046261s" totalImagesPullingTime="596.242665ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:26.648077371 +0000 UTC m=+1441.010897743" watchObservedRunningTime="2026-10-31 19:43:26.652046261 +0000 UTC m=+1441.014866625" Oct 31 19:43:29.690831 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:29.690790 4910 scope.go:122] "RemoveContainer" containerID="bd9b3693136852d4068e6cc9c75481166f672c8cc6b83e8a2e68b19d431c5e05" Oct 31 19:43:29.704343 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:29.704316 4910 scope.go:122] "RemoveContainer" containerID="c887a357871b0e70a14ed8ea3ffd43e889cd3817395b47c8f4327fa607212849" Oct 31 19:43:30.127506 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:30.127464 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1568f942-s7wjr"] Oct 31 19:43:30.129981 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:30.129956 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1568f942-s7wjr"] Oct 31 19:43:31.758028 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:31.757996 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ab18638-5a80-47e6-9ef0-02031de59836" path="/var/lib/kubelet/pods/3ab18638-5a80-47e6-9ef0-02031de59836/volumes" Oct 31 19:43:32.645092 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:32.645062 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8" Oct 31 19:43:41.689938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:41.689270 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7"] Oct 31 19:43:41.699969 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:41.695666 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1c0c1d-ztqw7"] Oct 31 19:43:41.803939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:41.779751 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f403772-348e-44cb-adf1-c695cf83f1ee" path="/var/lib/kubelet/pods/7f403772-348e-44cb-adf1-c695cf83f1ee/volumes" Oct 31 19:43:42.753421 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:43:42.753379 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:43:51.465937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:51.465678 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:51.732111 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:51.732025 4910 patch_prober.go:29] interesting pod/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:43:51.732285 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:51.732120 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" podUID="9f646f52-3c80-4fbb-8049-712f1e6b1595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:43:52.011276 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.011197 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz"] Oct 31 19:43:52.018685 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.018659 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.025467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.025441 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8dcfa8-openshift-default-dockercfg-jsqp5\"" Oct 31 19:43:52.061032 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.035157 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz"] Oct 31 19:43:52.134318 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.132991 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-token\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.134318 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.133207 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-credential-socket\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.134318 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.133333 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-envoy\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.134318 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.133453 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwqm9\" (UniqueName: \"kubernetes.io/projected/6da8e70c-0b96-48e1-92cb-01ca01434b98-kube-api-access-jwqm9\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.134318 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.133563 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da8e70c-0b96-48e1-92cb-01ca01434b98-istiod-ca-cert\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.134318 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.133597 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-workload-certs\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.134318 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.133709 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-podinfo\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.134318 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.133902 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-data\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.134318 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.133997 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-workload-socket\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.306035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.303792 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-data\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.306035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.303896 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-workload-socket\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.306035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.304076 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-token\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.306035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.304111 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-credential-socket\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.306035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.304163 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-envoy\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.306035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.304213 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwqm9\" (UniqueName: \"kubernetes.io/projected/6da8e70c-0b96-48e1-92cb-01ca01434b98-kube-api-access-jwqm9\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.306035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.304341 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da8e70c-0b96-48e1-92cb-01ca01434b98-istiod-ca-cert\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.306035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.304378 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-workload-certs\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.306035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.304403 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-podinfo\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.327936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.327189 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-credential-socket\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.327936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.327416 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-data\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.327936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.327622 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-workload-socket\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.360938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.344767 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da8e70c-0b96-48e1-92cb-01ca01434b98-istiod-ca-cert\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.360938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.345075 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-workload-certs\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.415803 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.415723 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-token\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.421063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.418623 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwqm9\" (UniqueName: \"kubernetes.io/projected/6da8e70c-0b96-48e1-92cb-01ca01434b98-kube-api-access-jwqm9\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.441781 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.441744 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-podinfo\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.473934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.468586 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb"] Oct 31 19:43:52.541933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.529603 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.545930 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.543340 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-785d26-openshift-default-dockercfg-5mwc5\"" Oct 31 19:43:52.638996 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.638958 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrtwm\" (UniqueName: \"kubernetes.io/projected/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-kube-api-access-zrtwm\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.639218 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.639196 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istiod-ca-cert\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.639300 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.639284 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-podinfo\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.639375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.639361 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-workload-socket\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.639432 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.639392 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-envoy\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.639529 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.639516 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-data\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.639678 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.639663 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-credential-socket\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.639741 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.639690 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-token\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.639798 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.639747 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-workload-certs\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.640738 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.640692 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-envoy\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.664125 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.663232 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb"] Oct 31 19:43:52.672156 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.671751 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:52.770011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.745316 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-data\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.770011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.745514 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-credential-socket\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.770011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.745546 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-token\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.770011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.745579 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-workload-certs\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.770011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.745695 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrtwm\" (UniqueName: \"kubernetes.io/projected/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-kube-api-access-zrtwm\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.770011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.745722 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istiod-ca-cert\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.770011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.745790 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-podinfo\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.770011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.745862 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-workload-socket\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.770011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.745892 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-envoy\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.770011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.758710 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-envoy\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.770011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.759038 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-data\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.770011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.759278 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-credential-socket\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.770011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.759869 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-workload-certs\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.770011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.760833 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istiod-ca-cert\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.770011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.763700 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-podinfo\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.770011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.763999 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-workload-socket\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.790357 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.784303 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-token\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:52.976758 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:52.964671 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrtwm\" (UniqueName: \"kubernetes.io/projected/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-kube-api-access-zrtwm\") pod \"e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:53.258981 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:53.253666 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:53.356932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:53.353421 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz"] Oct 31 19:43:53.359168 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:43:53.359116 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da8e70c_0b96_48e1_92cb_01ca01434b98.slice/crio-341a7b4dbc1eb682d80d084fa215a42ed6ca05838f9acf34718c21e00d8f337c WatchSource:0}: Error finding container 341a7b4dbc1eb682d80d084fa215a42ed6ca05838f9acf34718c21e00d8f337c: Status 404 returned error can't find the container with id 341a7b4dbc1eb682d80d084fa215a42ed6ca05838f9acf34718c21e00d8f337c Oct 31 19:43:53.384342 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:53.383626 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:43:53.384342 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:53.383735 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:43:53.384342 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:53.383777 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:43:53.723574 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:53.723528 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:43:53.724092 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:53.724062 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:43:53.724256 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:53.724236 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:43:53.737036 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:53.737004 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb"] Oct 31 19:43:53.746882 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:53.746848 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" event={"ID":"6da8e70c-0b96-48e1-92cb-01ca01434b98","Type":"ContainerStarted","Data":"33a0f003beabc7d5156e32eb35fff9331b705e3c20a2f984b0a1ee0e9d068749"} Oct 31 19:43:53.747118 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:53.747096 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" event={"ID":"6da8e70c-0b96-48e1-92cb-01ca01434b98","Type":"ContainerStarted","Data":"341a7b4dbc1eb682d80d084fa215a42ed6ca05838f9acf34718c21e00d8f337c"} Oct 31 19:43:54.032003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:54.031926 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" podStartSLOduration=3.031884568 podStartE2EDuration="3.031884568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:53.791327831 +0000 UTC m=+1468.154148204" watchObservedRunningTime="2026-10-31 19:43:54.031884568 +0000 UTC m=+1468.394704940" Oct 31 19:43:54.675181 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:54.675075 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:54.676678 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:54.676280 4910 patch_prober.go:29] interesting pod/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 19:43:54.676879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:54.676812 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" podUID="6da8e70c-0b96-48e1-92cb-01ca01434b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 19:43:54.764934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:54.764066 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" event={"ID":"7631819a-de14-4a11-a5a9-9b2e05e7c7a6","Type":"ContainerStarted","Data":"de17dab6e975433bab5a8de52bfc85dda1d4fa6131db47c3d80ed5afce1ab072"} Oct 31 19:43:54.764934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:54.764131 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" event={"ID":"7631819a-de14-4a11-a5a9-9b2e05e7c7a6","Type":"ContainerStarted","Data":"87bc256af108967066615400c2a7608bf2c2ab87b62f94da2ae9c5dbbb694c44"} Oct 31 19:43:54.791453 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:54.791389 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" podStartSLOduration=2.791368451 podStartE2EDuration="2.791368451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:54.790674776 +0000 UTC m=+1469.153495147" watchObservedRunningTime="2026-10-31 19:43:54.791368451 +0000 UTC m=+1469.154188825" Oct 31 19:43:55.255720 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:55.255685 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:55.264076 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:55.264041 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:55.689937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:55.686275 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:55.781641 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:55.778356 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:55.781641 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:55.779177 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:55.783204 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:55.783179 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:43:55.783977 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:55.783876 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:43:56.787563 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:56.787530 4910 generic.go:356] "Generic (PLEG): container finished" podID="9f646f52-3c80-4fbb-8049-712f1e6b1595" containerID="6f3b83ca6f6ad605de24d7751c2606b03ebbe6624508c6c6865d37a6e9523cbb" exitCode=0 Oct 31 19:43:56.788203 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:56.787614 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" event={"ID":"9f646f52-3c80-4fbb-8049-712f1e6b1595","Type":"ContainerDied","Data":"6f3b83ca6f6ad605de24d7751c2606b03ebbe6624508c6c6865d37a6e9523cbb"} Oct 31 19:43:56.788272 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:56.788254 4910 scope.go:122] "RemoveContainer" containerID="6f3b83ca6f6ad605de24d7751c2606b03ebbe6624508c6c6865d37a6e9523cbb" Oct 31 19:43:56.790385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:56.790351 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:43:56.790521 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:56.790439 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:43:56.790521 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:56.790492 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:43:57.814940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:57.810484 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" event={"ID":"9f646f52-3c80-4fbb-8049-712f1e6b1595","Type":"ContainerStarted","Data":"49b296095d99cebbcb1cf079e727995b87590d86300fbcf6e1c33012c4842652"} Oct 31 19:43:58.712187 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:58.712149 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:43:58.712608 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:58.712477 4910 patch_prober.go:29] interesting pod/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 31 19:43:58.712768 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:58.712742 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" podUID="9f646f52-3c80-4fbb-8049-712f1e6b1595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 31 19:43:59.705693 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:59.705650 4910 patch_prober.go:29] interesting pod/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 31 19:43:59.706242 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:43:59.705721 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" podUID="9f646f52-3c80-4fbb-8049-712f1e6b1595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 31 19:44:00.705468 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:00.705438 4910 patch_prober.go:29] interesting pod/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 31 19:44:00.705709 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:00.705682 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" podUID="9f646f52-3c80-4fbb-8049-712f1e6b1595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 31 19:44:01.705747 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:01.705705 4910 patch_prober.go:29] interesting pod/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 31 19:44:01.706284 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:01.705766 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" podUID="9f646f52-3c80-4fbb-8049-712f1e6b1595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 31 19:44:02.725006 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:02.724962 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d319a-2t84q"] Oct 31 19:44:02.749248 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:02.749214 4910 patch_prober.go:29] interesting pod/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 31 19:44:02.749552 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:02.749522 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" podUID="9f646f52-3c80-4fbb-8049-712f1e6b1595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 31 19:44:02.758401 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:02.758370 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d319a-2t84q"] Oct 31 19:44:03.060993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:03.054624 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0576ba46-rclk7"] Oct 31 19:44:03.071465 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:03.071432 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0576ba46-rclk7"] Oct 31 19:44:03.705817 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:03.705768 4910 patch_prober.go:29] interesting pod/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 31 19:44:03.706020 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:03.705843 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" podUID="9f646f52-3c80-4fbb-8049-712f1e6b1595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 31 19:44:03.764200 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:03.764165 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d191ec1-875b-4dc3-a3f9-e896d53dc00f" path="/var/lib/kubelet/pods/4d191ec1-875b-4dc3-a3f9-e896d53dc00f/volumes" Oct 31 19:44:03.765423 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:03.765398 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50252ae6-f1ab-4a90-a33d-36b0ed477a49" path="/var/lib/kubelet/pods/50252ae6-f1ab-4a90-a33d-36b0ed477a49/volumes" Oct 31 19:44:04.705567 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:04.705530 4910 patch_prober.go:29] interesting pod/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 31 19:44:04.705866 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:04.705826 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" podUID="9f646f52-3c80-4fbb-8049-712f1e6b1595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 31 19:44:05.682823 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:05.682793 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:44:05.711184 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:05.711152 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:44:06.173313 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:06.173280 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:44:09.035450 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:09.035412 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d16ace94-jqddp"] Oct 31 19:44:09.041681 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:09.041650 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d16ace94-jqddp"] Oct 31 19:44:09.767971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:09.767937 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ebff398-c9da-4829-910b-497908ea7feb" path="/var/lib/kubelet/pods/0ebff398-c9da-4829-910b-497908ea7feb/volumes" Oct 31 19:44:10.564532 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:10.564489 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q"] Oct 31 19:44:10.570514 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:10.570493 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q" Oct 31 19:44:10.581408 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:10.581383 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q"] Oct 31 19:44:10.595507 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:10.595476 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:44:10.595655 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:10.595596 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:44:10.604618 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:10.604588 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb47a2d-9025-4800-bf5d-85e8a881c4f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q\" (UID: \"7bb47a2d-9025-4800-bf5d-85e8a881c4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q" Oct 31 19:44:10.604766 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:10.604634 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt5d8\" (UniqueName: \"kubernetes.io/projected/7bb47a2d-9025-4800-bf5d-85e8a881c4f1-kube-api-access-xt5d8\") pod \"maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q\" (UID: \"7bb47a2d-9025-4800-bf5d-85e8a881c4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q" Oct 31 19:44:10.604837 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:10.604767 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb47a2d-9025-4800-bf5d-85e8a881c4f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q\" (UID: \"7bb47a2d-9025-4800-bf5d-85e8a881c4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q" Oct 31 19:44:10.706661 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:10.706621 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb47a2d-9025-4800-bf5d-85e8a881c4f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q\" (UID: \"7bb47a2d-9025-4800-bf5d-85e8a881c4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q" Oct 31 19:44:10.706875 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:10.706676 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt5d8\" (UniqueName: \"kubernetes.io/projected/7bb47a2d-9025-4800-bf5d-85e8a881c4f1-kube-api-access-xt5d8\") pod \"maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q\" (UID: \"7bb47a2d-9025-4800-bf5d-85e8a881c4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q" Oct 31 19:44:10.706875 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:10.706841 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb47a2d-9025-4800-bf5d-85e8a881c4f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q\" (UID: \"7bb47a2d-9025-4800-bf5d-85e8a881c4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q" Oct 31 19:44:10.707621 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:10.707594 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb47a2d-9025-4800-bf5d-85e8a881c4f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q\" (UID: \"7bb47a2d-9025-4800-bf5d-85e8a881c4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q" Oct 31 19:44:10.709293 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:10.709267 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb47a2d-9025-4800-bf5d-85e8a881c4f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q\" (UID: \"7bb47a2d-9025-4800-bf5d-85e8a881c4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q" Oct 31 19:44:10.720438 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:10.720411 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt5d8\" (UniqueName: \"kubernetes.io/projected/7bb47a2d-9025-4800-bf5d-85e8a881c4f1-kube-api-access-xt5d8\") pod \"maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q\" (UID: \"7bb47a2d-9025-4800-bf5d-85e8a881c4f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q" Oct 31 19:44:10.895719 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:10.895612 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q" Oct 31 19:44:11.205351 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:44:11.205318 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb47a2d_9025_4800_bf5d_85e8a881c4f1.slice/crio-0036bd77f4597fcadf65edb764cb98dba64ae863d876018642398746ba510c7a WatchSource:0}: Error finding container 0036bd77f4597fcadf65edb764cb98dba64ae863d876018642398746ba510c7a: Status 404 returned error can't find the container with id 0036bd77f4597fcadf65edb764cb98dba64ae863d876018642398746ba510c7a Oct 31 19:44:11.206179 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:11.206153 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q"] Oct 31 19:44:11.227474 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:11.227442 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q" event={"ID":"7bb47a2d-9025-4800-bf5d-85e8a881c4f1","Type":"ContainerStarted","Data":"0036bd77f4597fcadf65edb764cb98dba64ae863d876018642398746ba510c7a"} Oct 31 19:44:12.239255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:12.239213 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q" event={"ID":"7bb47a2d-9025-4800-bf5d-85e8a881c4f1","Type":"ContainerStarted","Data":"2bd28068e2b8a12ee4ef46d2c10a26b4cfcc1ce391039696eb9433873539dafc"} Oct 31 19:44:12.265256 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:12.265205 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q" podStartSLOduration=2.265190788 podStartE2EDuration="2.265190788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:44:12.260865753 +0000 UTC m=+1486.623686126" watchObservedRunningTime="2026-10-31 19:44:12.265190788 +0000 UTC m=+1486.628011180" Oct 31 19:44:12.715453 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:44:12.715409 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:44:13.251054 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:13.251023 4910 generic.go:356] "Generic (PLEG): container finished" podID="7bb47a2d-9025-4800-bf5d-85e8a881c4f1" containerID="2bd28068e2b8a12ee4ef46d2c10a26b4cfcc1ce391039696eb9433873539dafc" exitCode=0 Oct 31 19:44:13.251440 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:13.251107 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q" event={"ID":"7bb47a2d-9025-4800-bf5d-85e8a881c4f1","Type":"ContainerDied","Data":"2bd28068e2b8a12ee4ef46d2c10a26b4cfcc1ce391039696eb9433873539dafc"} Oct 31 19:44:14.446119 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:14.446082 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q" Oct 31 19:44:14.517098 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:14.517011 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bb47a2d-9025-4800-bf5d-85e8a881c4f1-kube-api-access-xt5d8\" (UniqueName: \"kubernetes.io/projected/7bb47a2d-9025-4800-bf5d-85e8a881c4f1-kube-api-access-xt5d8\") pod \"7bb47a2d-9025-4800-bf5d-85e8a881c4f1\" (UID: \"7bb47a2d-9025-4800-bf5d-85e8a881c4f1\") " Oct 31 19:44:14.517509 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:14.517484 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bb47a2d-9025-4800-bf5d-85e8a881c4f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb47a2d-9025-4800-bf5d-85e8a881c4f1-maas-api-openshift-ca\") pod \"7bb47a2d-9025-4800-bf5d-85e8a881c4f1\" (UID: \"7bb47a2d-9025-4800-bf5d-85e8a881c4f1\") " Oct 31 19:44:14.517661 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:14.517569 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bb47a2d-9025-4800-bf5d-85e8a881c4f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb47a2d-9025-4800-bf5d-85e8a881c4f1-maas-api-xks-ca\") pod \"7bb47a2d-9025-4800-bf5d-85e8a881c4f1\" (UID: \"7bb47a2d-9025-4800-bf5d-85e8a881c4f1\") " Oct 31 19:44:14.518141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:14.518094 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb47a2d-9025-4800-bf5d-85e8a881c4f1-maas-api-openshift-ca" pod "7bb47a2d-9025-4800-bf5d-85e8a881c4f1" (UID: "7bb47a2d-9025-4800-bf5d-85e8a881c4f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:44:14.522640 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:14.522611 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb47a2d-9025-4800-bf5d-85e8a881c4f1-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:44:14.523597 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:14.523547 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bb47a2d-9025-4800-bf5d-85e8a881c4f1-maas-api-xks-ca" pod "7bb47a2d-9025-4800-bf5d-85e8a881c4f1" (UID: "7bb47a2d-9025-4800-bf5d-85e8a881c4f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:44:14.523597 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:14.523535 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb47a2d-9025-4800-bf5d-85e8a881c4f1-kube-api-access-xt5d8" pod "7bb47a2d-9025-4800-bf5d-85e8a881c4f1" (UID: "7bb47a2d-9025-4800-bf5d-85e8a881c4f1"). InnerVolumeSpecName "kube-api-access-xt5d8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:44:14.624695 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:14.624657 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt5d8\" (UniqueName: \"kubernetes.io/projected/7bb47a2d-9025-4800-bf5d-85e8a881c4f1-kube-api-access-xt5d8\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:44:14.624695 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:14.624697 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb47a2d-9025-4800-bf5d-85e8a881c4f1-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:44:15.272556 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:15.272530 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q" Oct 31 19:44:15.272735 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:15.272576 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q" event={"ID":"7bb47a2d-9025-4800-bf5d-85e8a881c4f1","Type":"ContainerDied","Data":"0036bd77f4597fcadf65edb764cb98dba64ae863d876018642398746ba510c7a"} Oct 31 19:44:15.272735 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:15.272626 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0036bd77f4597fcadf65edb764cb98dba64ae863d876018642398746ba510c7a" Oct 31 19:44:24.389807 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:24.389773 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj"] Oct 31 19:44:24.392582 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:24.392539 4910 cpu_manager.go:414] "containerMap: removing container" podUID="7bb47a2d-9025-4800-bf5d-85e8a881c4f1" containerName="revoke-keys" Oct 31 19:44:24.392582 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:24.392571 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bb47a2d-9025-4800-bf5d-85e8a881c4f1" containerName="revoke-keys" Oct 31 19:44:24.392802 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:24.392782 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bb47a2d-9025-4800-bf5d-85e8a881c4f1" containerName="revoke-keys" Oct 31 19:44:24.399073 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:24.399045 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj" Oct 31 19:44:24.408479 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:24.407356 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj"] Oct 31 19:44:24.414625 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:24.414409 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-785d26-serving-cert\"" Oct 31 19:44:24.533382 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:24.533344 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59df105-1b29-4a08-8677-69af26994de8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj\" (UID: \"d59df105-1b29-4a08-8677-69af26994de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj" Oct 31 19:44:24.533582 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:24.533392 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5r5h\" (UniqueName: \"kubernetes.io/projected/d59df105-1b29-4a08-8677-69af26994de8-kube-api-access-m5r5h\") pod \"maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj\" (UID: \"d59df105-1b29-4a08-8677-69af26994de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj" Oct 31 19:44:24.533644 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:24.533602 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d59df105-1b29-4a08-8677-69af26994de8-maas-api-tls\") pod \"maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj\" (UID: \"d59df105-1b29-4a08-8677-69af26994de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj" Oct 31 19:44:24.533722 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:24.533709 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59df105-1b29-4a08-8677-69af26994de8-metrics-certs\") pod \"maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj\" (UID: \"d59df105-1b29-4a08-8677-69af26994de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj" Oct 31 19:44:24.645986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:24.638832 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59df105-1b29-4a08-8677-69af26994de8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj\" (UID: \"d59df105-1b29-4a08-8677-69af26994de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj" Oct 31 19:44:24.645986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:24.638898 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5r5h\" (UniqueName: \"kubernetes.io/projected/d59df105-1b29-4a08-8677-69af26994de8-kube-api-access-m5r5h\") pod \"maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj\" (UID: \"d59df105-1b29-4a08-8677-69af26994de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj" Oct 31 19:44:24.645986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:24.639084 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d59df105-1b29-4a08-8677-69af26994de8-maas-api-tls\") pod \"maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj\" (UID: \"d59df105-1b29-4a08-8677-69af26994de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj" Oct 31 19:44:24.645986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:24.639190 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59df105-1b29-4a08-8677-69af26994de8-metrics-certs\") pod \"maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj\" (UID: \"d59df105-1b29-4a08-8677-69af26994de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj" Oct 31 19:44:24.660929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:24.652302 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59df105-1b29-4a08-8677-69af26994de8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj\" (UID: \"d59df105-1b29-4a08-8677-69af26994de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj" Oct 31 19:44:24.681992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:24.667091 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59df105-1b29-4a08-8677-69af26994de8-metrics-certs\") pod \"maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj\" (UID: \"d59df105-1b29-4a08-8677-69af26994de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj" Oct 31 19:44:24.696773 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:24.696699 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5r5h\" (UniqueName: \"kubernetes.io/projected/d59df105-1b29-4a08-8677-69af26994de8-kube-api-access-m5r5h\") pod \"maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj\" (UID: \"d59df105-1b29-4a08-8677-69af26994de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj" Oct 31 19:44:24.705956 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:24.697516 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d59df105-1b29-4a08-8677-69af26994de8-maas-api-tls\") pod \"maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj\" (UID: \"d59df105-1b29-4a08-8677-69af26994de8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj" Oct 31 19:44:24.731311 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:24.727463 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj" Oct 31 19:44:25.054711 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:25.054669 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj"] Oct 31 19:44:25.056603 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:44:25.056566 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59df105_1b29_4a08_8677_69af26994de8.slice/crio-9b0918b954cd1267e11f1db6b59b6d1db93d53db26318dcdbc21db2e342c6e74 WatchSource:0}: Error finding container 9b0918b954cd1267e11f1db6b59b6d1db93d53db26318dcdbc21db2e342c6e74: Status 404 returned error can't find the container with id 9b0918b954cd1267e11f1db6b59b6d1db93d53db26318dcdbc21db2e342c6e74 Oct 31 19:44:25.439603 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:25.439556 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj" event={"ID":"d59df105-1b29-4a08-8677-69af26994de8","Type":"ContainerStarted","Data":"9b0918b954cd1267e11f1db6b59b6d1db93d53db26318dcdbc21db2e342c6e74"} Oct 31 19:44:26.194301 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:26.194265 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 19:44:26.251252 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:26.251222 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 19:44:26.451786 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:26.451696 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj" event={"ID":"d59df105-1b29-4a08-8677-69af26994de8","Type":"ContainerStarted","Data":"a72ce22618558c704de7125c3b7b380f86c9620e0d3187722f9a10be1c97da7f"} Oct 31 19:44:26.452208 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:26.451842 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj" Oct 31 19:44:26.478339 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:26.478285 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj" podStartSLOduration=2.020895278 podStartE2EDuration="2.478266637s" totalImagesPullingTime="457.371359ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:44:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:44:26.475083299 +0000 UTC m=+1500.837903671" watchObservedRunningTime="2026-10-31 19:44:26.478266637 +0000 UTC m=+1500.841087011" Oct 31 19:44:29.804978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:29.804948 4910 scope.go:122] "RemoveContainer" containerID="90a8064849484831733d01b8c0bbdbd104a593b72722b4c86c1ab824d6b6c185" Oct 31 19:44:32.472395 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:32.472363 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj" Oct 31 19:44:41.357935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:41.330182 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:44:41.473933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:41.440063 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:44:41.473933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:41.467434 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:44:41.549987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:41.531016 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:44:41.575474 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:41.569154 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:44:41.746990 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:41.741886 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:44:41.912355 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:41.890455 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:44:41.912573 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:41.912523 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:44:42.143395 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:42.143346 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:44:42.166543 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:42.166493 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:44:42.237305 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:42.237254 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:44:42.344306 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:42.344256 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:44:42.377408 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:42.347287 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 19:44:42.378879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:42.360515 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded" Oct 31 19:44:42.378879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:42.361632 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:44:42.379208 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:42.379186 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:44:42.391735 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:42.391699 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:44:42.442106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:42.442012 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:44:42.455256 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:42.442757 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:44:42.579938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:42.555158 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:44:42.581622 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:42.581590 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:44:42.601205 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:42.576021 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:44:42.602531 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:42.601896 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:44:42.766303 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:42.761343 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:44:42.774449 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:42.773964 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:44:42.796428 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:42.776655 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:44:42.797031 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:42.797000 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:44:42.900052 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:42.900002 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:44:43.205938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:42.913999 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:44:43.261942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:43.227006 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 19:44:43.261942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:42.914082 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 19:44:43.289800 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:43.289768 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:44:43.292738 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:43.292714 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:44:43.292987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:43.292971 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:44:43.384047 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:43.384002 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:44:43.400855 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:43.394465 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:44:43.465281 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:43.438937 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:44:43.511937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:43.449981 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:44:43.578933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:43.570723 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:44:43.578933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:43.572485 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:44:43.580584 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:43.580546 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:44:43.612058 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:43.612012 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 19:44:43.627932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:43.615397 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:44:43.676870 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:43.676840 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:44:43.714184 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:44:43.710419 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:44:43.766015 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:43.765931 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:44:44.002320 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:44.002290 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:44:44.002644 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:44.002624 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 19:44:44.014461 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:44.014408 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:44:44.014699 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:44.014682 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:44:44.014873 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:44.014860 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:44:44.015118 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:44.015098 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:44:44.015371 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:44.015350 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:44:53.826088 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:53.826036 4910 patch_prober.go:29] interesting pod/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:44:53.826606 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:53.826112 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" podUID="5e8d77e7-9866-404f-83fc-1809898cec90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:44:54.428183 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.428103 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5"] Oct 31 19:44:54.433249 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.433218 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:54.458517 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.448866 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5"] Oct 31 19:44:54.458517 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.457482 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d8c398-openshift-default-dockercfg-6tdvv\"" Oct 31 19:44:54.663020 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.603535 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-workload-certs\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:54.663020 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.603597 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-istio-data\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:54.663020 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.603688 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-workload-socket\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:54.663020 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.603721 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27663054-5fbb-4a52-b101-682623a5f518-istio-token\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:54.663020 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.603872 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27663054-5fbb-4a52-b101-682623a5f518-istio-podinfo\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:54.663020 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.604063 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9kjr\" (UniqueName: \"kubernetes.io/projected/27663054-5fbb-4a52-b101-682623a5f518-kube-api-access-r9kjr\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:54.663020 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.604164 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-credential-socket\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:54.663020 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.604276 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-istio-envoy\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:54.663020 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.604335 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27663054-5fbb-4a52-b101-682623a5f518-istiod-ca-cert\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:54.733964 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.708221 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-istio-data\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:54.733964 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.708351 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-workload-socket\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:54.733964 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.708394 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27663054-5fbb-4a52-b101-682623a5f518-istio-token\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:54.733964 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.708521 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27663054-5fbb-4a52-b101-682623a5f518-istio-podinfo\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:54.733964 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.708648 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9kjr\" (UniqueName: \"kubernetes.io/projected/27663054-5fbb-4a52-b101-682623a5f518-kube-api-access-r9kjr\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:54.733964 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.708712 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-credential-socket\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:54.733964 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.708794 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-istio-envoy\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:54.733964 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.708846 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27663054-5fbb-4a52-b101-682623a5f518-istiod-ca-cert\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:54.790204 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.747514 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-workload-socket\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:54.790204 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.747760 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-istio-data\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:54.790204 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.747997 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-credential-socket\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:54.828654 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.791283 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27663054-5fbb-4a52-b101-682623a5f518-istiod-ca-cert\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:54.828654 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.826687 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9kjr\" (UniqueName: \"kubernetes.io/projected/27663054-5fbb-4a52-b101-682623a5f518-kube-api-access-r9kjr\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:54.829477 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.829427 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-workload-certs\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:54.859717 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.859648 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-workload-certs\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:55.030329 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.966554 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27663054-5fbb-4a52-b101-682623a5f518-istio-token\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:55.030329 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:54.992702 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-istio-envoy\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:55.030329 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:55.010523 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27663054-5fbb-4a52-b101-682623a5f518-istio-podinfo\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:55.100920 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:55.100846 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:55.102728 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:55.102661 4910 generic.go:356] "Generic (PLEG): container finished" podID="5e8d77e7-9866-404f-83fc-1809898cec90" containerID="887e5ff8622f885dc948a0d730a1d9c5e85176dbf723a89aaf4433bfd83853b6" exitCode=0 Oct 31 19:44:55.106765 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:55.103775 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" event={"ID":"5e8d77e7-9866-404f-83fc-1809898cec90","Type":"ContainerDied","Data":"887e5ff8622f885dc948a0d730a1d9c5e85176dbf723a89aaf4433bfd83853b6"} Oct 31 19:44:55.128808 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:55.128734 4910 scope.go:122] "RemoveContainer" containerID="887e5ff8622f885dc948a0d730a1d9c5e85176dbf723a89aaf4433bfd83853b6" Oct 31 19:44:55.132637 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:55.132583 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:44:55.132933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:55.132750 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:44:55.132933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:55.132858 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:44:55.556119 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:55.550435 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5"] Oct 31 19:44:55.560413 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:55.560368 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:44:55.560537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:55.560464 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:44:55.560537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:55.560509 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:44:56.134635 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:56.134589 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" event={"ID":"27663054-5fbb-4a52-b101-682623a5f518","Type":"ContainerStarted","Data":"cd393767ffcf2b3f30129e3923bb28f97ad899ff30836b6c05c36f1dd0d7f536"} Oct 31 19:44:56.135709 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:56.135680 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" event={"ID":"27663054-5fbb-4a52-b101-682623a5f518","Type":"ContainerStarted","Data":"885bb4eb6cebd5f642d3dd20cddc9c316deab2c08d56811156aa6f3b05c197a5"} Oct 31 19:44:56.159058 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:56.159013 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" event={"ID":"5e8d77e7-9866-404f-83fc-1809898cec90","Type":"ContainerStarted","Data":"241c1ff159899727ff021baf6d388074d37d7e94d525a6520f56d1bb22a8195b"} Oct 31 19:44:56.178233 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:56.178173 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" podStartSLOduration=2.178139537 podStartE2EDuration="2.178139537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:44:56.176522572 +0000 UTC m=+1530.539342944" watchObservedRunningTime="2026-10-31 19:44:56.178139537 +0000 UTC m=+1530.540959914" Oct 31 19:44:57.094671 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:57.094632 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:44:57.094864 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:57.094819 4910 patch_prober.go:29] interesting pod/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 31 19:44:57.094964 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:57.094892 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" podUID="5e8d77e7-9866-404f-83fc-1809898cec90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 31 19:44:57.101576 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:57.101550 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:57.108423 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:57.108405 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:57.172725 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:57.172687 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:57.173742 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:57.173721 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:44:58.095262 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:58.095225 4910 patch_prober.go:29] interesting pod/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 31 19:44:58.095477 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:58.095284 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" podUID="5e8d77e7-9866-404f-83fc-1809898cec90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 31 19:44:59.094546 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:59.094509 4910 patch_prober.go:29] interesting pod/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 31 19:44:59.095106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:44:59.094566 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" podUID="5e8d77e7-9866-404f-83fc-1809898cec90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 31 19:45:00.100817 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.100779 4910 patch_prober.go:29] interesting pod/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 31 19:45:00.107214 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.100850 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" podUID="5e8d77e7-9866-404f-83fc-1809898cec90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 31 19:45:00.448052 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.447954 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq"] Oct 31 19:45:00.474450 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.465301 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq" Oct 31 19:45:00.532940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.526317 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:45:00.617388 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.615094 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mvx2\" (UniqueName: \"kubernetes.io/projected/f1cfaf0d-c80e-4860-828d-7ea807d34ccc-kube-api-access-5mvx2\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq\" (UID: \"f1cfaf0d-c80e-4860-828d-7ea807d34ccc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq" Oct 31 19:45:00.617388 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.615258 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1cfaf0d-c80e-4860-828d-7ea807d34ccc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq\" (UID: \"f1cfaf0d-c80e-4860-828d-7ea807d34ccc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq" Oct 31 19:45:00.699533 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.699442 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp"] Oct 31 19:45:00.706608 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.706582 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp" Oct 31 19:45:00.711024 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.709543 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2"] Oct 31 19:45:00.719470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.718990 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mvx2\" (UniqueName: \"kubernetes.io/projected/f1cfaf0d-c80e-4860-828d-7ea807d34ccc-kube-api-access-5mvx2\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq\" (UID: \"f1cfaf0d-c80e-4860-828d-7ea807d34ccc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq" Oct 31 19:45:00.723210 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.719742 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1cfaf0d-c80e-4860-828d-7ea807d34ccc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq\" (UID: \"f1cfaf0d-c80e-4860-828d-7ea807d34ccc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq" Oct 31 19:45:00.723210 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.721475 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1cfaf0d-c80e-4860-828d-7ea807d34ccc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq\" (UID: \"f1cfaf0d-c80e-4860-828d-7ea807d34ccc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq" Oct 31 19:45:00.741009 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.740969 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2" Oct 31 19:45:00.833282 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.796655 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq"] Oct 31 19:45:00.878994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.863892 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k78xh\" (UniqueName: \"kubernetes.io/projected/c52c8a23-5d17-4b66-995e-beb28777c843-kube-api-access-k78xh\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2\" (UID: \"c52c8a23-5d17-4b66-995e-beb28777c843\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2" Oct 31 19:45:00.878994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.863998 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7hvb\" (UniqueName: \"kubernetes.io/projected/64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab-kube-api-access-l7hvb\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp\" (UID: \"64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp" Oct 31 19:45:00.878994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.864342 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c52c8a23-5d17-4b66-995e-beb28777c843-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2\" (UID: \"c52c8a23-5d17-4b66-995e-beb28777c843\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2" Oct 31 19:45:00.878994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.864426 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp\" (UID: \"64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp" Oct 31 19:45:00.886019 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.885990 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp"] Oct 31 19:45:00.888015 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.887983 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mvx2\" (UniqueName: \"kubernetes.io/projected/f1cfaf0d-c80e-4860-828d-7ea807d34ccc-kube-api-access-5mvx2\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq\" (UID: \"f1cfaf0d-c80e-4860-828d-7ea807d34ccc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq" Oct 31 19:45:00.893191 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.893133 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb"] Oct 31 19:45:00.903468 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.903446 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb" Oct 31 19:45:00.903656 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.903632 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd"] Oct 31 19:45:00.925945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.922185 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd" Oct 31 19:45:00.958261 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.958181 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq" Oct 31 19:45:00.960656 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.960558 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb"] Oct 31 19:45:00.967010 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.966973 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k78xh\" (UniqueName: \"kubernetes.io/projected/c52c8a23-5d17-4b66-995e-beb28777c843-kube-api-access-k78xh\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2\" (UID: \"c52c8a23-5d17-4b66-995e-beb28777c843\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2" Oct 31 19:45:00.967302 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.967256 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7hvb\" (UniqueName: \"kubernetes.io/projected/64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab-kube-api-access-l7hvb\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp\" (UID: \"64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp" Oct 31 19:45:00.967504 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.967483 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6e841b0-383b-41dc-9d35-05f57d92ea1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb\" (UID: \"b6e841b0-383b-41dc-9d35-05f57d92ea1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb" Oct 31 19:45:00.967736 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.967714 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c52c8a23-5d17-4b66-995e-beb28777c843-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2\" (UID: \"c52c8a23-5d17-4b66-995e-beb28777c843\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2" Oct 31 19:45:00.967972 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.967901 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp\" (UID: \"64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp" Oct 31 19:45:00.968296 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.968272 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lhzt\" (UniqueName: \"kubernetes.io/projected/b6e841b0-383b-41dc-9d35-05f57d92ea1d-kube-api-access-4lhzt\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb\" (UID: \"b6e841b0-383b-41dc-9d35-05f57d92ea1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb" Oct 31 19:45:00.968891 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.968867 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c52c8a23-5d17-4b66-995e-beb28777c843-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2\" (UID: \"c52c8a23-5d17-4b66-995e-beb28777c843\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2" Oct 31 19:45:00.969294 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.969272 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp\" (UID: \"64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp" Oct 31 19:45:00.971853 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.971836 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd"] Oct 31 19:45:00.982632 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.982610 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7hvb\" (UniqueName: \"kubernetes.io/projected/64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab-kube-api-access-l7hvb\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp\" (UID: \"64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp" Oct 31 19:45:00.984372 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.984350 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2"] Oct 31 19:45:00.984494 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:00.984419 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k78xh\" (UniqueName: \"kubernetes.io/projected/c52c8a23-5d17-4b66-995e-beb28777c843-kube-api-access-k78xh\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2\" (UID: \"c52c8a23-5d17-4b66-995e-beb28777c843\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2" Oct 31 19:45:01.096839 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:01.078027 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm44v\" (UniqueName: \"kubernetes.io/projected/934b6146-1f28-42ae-8d56-46e6dfcae403-kube-api-access-xm44v\") pod \"maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd\" (UID: \"934b6146-1f28-42ae-8d56-46e6dfcae403\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd" Oct 31 19:45:01.096839 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:01.078181 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6e841b0-383b-41dc-9d35-05f57d92ea1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb\" (UID: \"b6e841b0-383b-41dc-9d35-05f57d92ea1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb" Oct 31 19:45:01.096839 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:01.078285 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/934b6146-1f28-42ae-8d56-46e6dfcae403-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd\" (UID: \"934b6146-1f28-42ae-8d56-46e6dfcae403\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd" Oct 31 19:45:01.096839 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:01.078483 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lhzt\" (UniqueName: \"kubernetes.io/projected/b6e841b0-383b-41dc-9d35-05f57d92ea1d-kube-api-access-4lhzt\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb\" (UID: \"b6e841b0-383b-41dc-9d35-05f57d92ea1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb" Oct 31 19:45:01.096839 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:01.079764 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6e841b0-383b-41dc-9d35-05f57d92ea1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb\" (UID: \"b6e841b0-383b-41dc-9d35-05f57d92ea1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb" Oct 31 19:45:01.102898 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:01.102867 4910 patch_prober.go:29] interesting pod/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 31 19:45:01.103422 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:01.102982 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" podUID="5e8d77e7-9866-404f-83fc-1809898cec90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 31 19:45:01.108806 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:01.106931 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lhzt\" (UniqueName: \"kubernetes.io/projected/b6e841b0-383b-41dc-9d35-05f57d92ea1d-kube-api-access-4lhzt\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb\" (UID: \"b6e841b0-383b-41dc-9d35-05f57d92ea1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb" Oct 31 19:45:01.180644 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:01.180016 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm44v\" (UniqueName: \"kubernetes.io/projected/934b6146-1f28-42ae-8d56-46e6dfcae403-kube-api-access-xm44v\") pod \"maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd\" (UID: \"934b6146-1f28-42ae-8d56-46e6dfcae403\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd" Oct 31 19:45:01.180644 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:01.180221 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/934b6146-1f28-42ae-8d56-46e6dfcae403-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd\" (UID: \"934b6146-1f28-42ae-8d56-46e6dfcae403\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd" Oct 31 19:45:01.181370 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:01.181297 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/934b6146-1f28-42ae-8d56-46e6dfcae403-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd\" (UID: \"934b6146-1f28-42ae-8d56-46e6dfcae403\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd" Oct 31 19:45:01.196474 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:01.195980 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm44v\" (UniqueName: \"kubernetes.io/projected/934b6146-1f28-42ae-8d56-46e6dfcae403-kube-api-access-xm44v\") pod \"maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd\" (UID: \"934b6146-1f28-42ae-8d56-46e6dfcae403\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd" Oct 31 19:45:01.196474 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:01.196272 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp" Oct 31 19:45:01.202309 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:01.202281 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2" Oct 31 19:45:01.274444 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:01.274414 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb" Oct 31 19:45:01.288170 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:01.288139 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd" Oct 31 19:45:01.402360 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:45:01.402303 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1cfaf0d_c80e_4860_828d_7ea807d34ccc.slice/crio-3c1f1026352cb0d68b2a32bf8542618e5fe1facb0436dcc4fe828090794f3c3b WatchSource:0}: Error finding container 3c1f1026352cb0d68b2a32bf8542618e5fe1facb0436dcc4fe828090794f3c3b: Status 404 returned error can't find the container with id 3c1f1026352cb0d68b2a32bf8542618e5fe1facb0436dcc4fe828090794f3c3b Oct 31 19:45:01.412577 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:01.412547 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq"] Oct 31 19:45:01.498979 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:01.498409 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq" event={"ID":"f1cfaf0d-c80e-4860-828d-7ea807d34ccc","Type":"ContainerStarted","Data":"3c1f1026352cb0d68b2a32bf8542618e5fe1facb0436dcc4fe828090794f3c3b"} Oct 31 19:45:01.952097 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:01.951856 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp"] Oct 31 19:45:01.969725 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:01.969696 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2"] Oct 31 19:45:01.974477 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:45:01.974447 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod934b6146_1f28_42ae_8d56_46e6dfcae403.slice/crio-79e9c40d0ff5afe05edce6e8054f9880ce5205ad7921ffc7daff88304a7247aa WatchSource:0}: Error finding container 79e9c40d0ff5afe05edce6e8054f9880ce5205ad7921ffc7daff88304a7247aa: Status 404 returned error can't find the container with id 79e9c40d0ff5afe05edce6e8054f9880ce5205ad7921ffc7daff88304a7247aa Oct 31 19:45:01.990655 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:01.986291 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb"] Oct 31 19:45:01.991784 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:01.991751 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd"] Oct 31 19:45:02.095135 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:02.095058 4910 patch_prober.go:29] interesting pod/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 31 19:45:02.095278 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:02.095122 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" podUID="5e8d77e7-9866-404f-83fc-1809898cec90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 31 19:45:02.516030 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:02.515991 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb" event={"ID":"b6e841b0-383b-41dc-9d35-05f57d92ea1d","Type":"ContainerStarted","Data":"3572ccec29e225c165e5aaa946ea5703be94e14f23c92d13849264595fd0d41b"} Oct 31 19:45:02.523758 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:02.523723 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp" event={"ID":"64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab","Type":"ContainerStarted","Data":"48ec2f808cb68439c6a277737f0f89ec614aca98f880699e9e6dea9365ffe122"} Oct 31 19:45:02.540721 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:02.540689 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq" event={"ID":"f1cfaf0d-c80e-4860-828d-7ea807d34ccc","Type":"ContainerStarted","Data":"8261ec38270042a62b232a0b9a6f7a98412a9da99f173df9e6ff035a2ee1eb0f"} Oct 31 19:45:02.550346 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:02.550311 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2" event={"ID":"c52c8a23-5d17-4b66-995e-beb28777c843","Type":"ContainerStarted","Data":"dbf5fb25c9bc5a00b366787eea51c2a021ae791ce28a6e3dc1b79c3846f97e31"} Oct 31 19:45:02.552140 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:02.552117 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd" event={"ID":"934b6146-1f28-42ae-8d56-46e6dfcae403","Type":"ContainerStarted","Data":"79e9c40d0ff5afe05edce6e8054f9880ce5205ad7921ffc7daff88304a7247aa"} Oct 31 19:45:02.560627 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:02.560576 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq" podStartSLOduration=2.560560838 podStartE2EDuration="2.560560838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:02.555225404 +0000 UTC m=+1536.918045776" watchObservedRunningTime="2026-10-31 19:45:02.560560838 +0000 UTC m=+1536.923381212" Oct 31 19:45:03.113461 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:03.113423 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:45:03.115721 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:03.115695 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:45:03.116808 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:03.116784 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:45:03.564356 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:03.564321 4910 generic.go:356] "Generic (PLEG): container finished" podID="f1cfaf0d-c80e-4860-828d-7ea807d34ccc" containerID="8261ec38270042a62b232a0b9a6f7a98412a9da99f173df9e6ff035a2ee1eb0f" exitCode=60 Oct 31 19:45:03.564780 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:03.564413 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq" event={"ID":"f1cfaf0d-c80e-4860-828d-7ea807d34ccc","Type":"ContainerDied","Data":"8261ec38270042a62b232a0b9a6f7a98412a9da99f173df9e6ff035a2ee1eb0f"} Oct 31 19:45:03.564780 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:03.564735 4910 scope.go:122] "RemoveContainer" containerID="8261ec38270042a62b232a0b9a6f7a98412a9da99f173df9e6ff035a2ee1eb0f" Oct 31 19:45:03.566821 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:03.566798 4910 generic.go:356] "Generic (PLEG): container finished" podID="c52c8a23-5d17-4b66-995e-beb28777c843" containerID="66de7e52102798889f25ef0d6721a1398fac32f61b1fa5c62c01bb08956ea802" exitCode=60 Oct 31 19:45:03.566959 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:03.566881 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2" event={"ID":"c52c8a23-5d17-4b66-995e-beb28777c843","Type":"ContainerDied","Data":"66de7e52102798889f25ef0d6721a1398fac32f61b1fa5c62c01bb08956ea802"} Oct 31 19:45:03.567187 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:03.567169 4910 scope.go:122] "RemoveContainer" containerID="66de7e52102798889f25ef0d6721a1398fac32f61b1fa5c62c01bb08956ea802" Oct 31 19:45:03.569755 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:03.569443 4910 generic.go:356] "Generic (PLEG): container finished" podID="934b6146-1f28-42ae-8d56-46e6dfcae403" containerID="2e115922703e402b76626395fb1d7b3fb04caa372f65ecd0f651c7099734a5e4" exitCode=60 Oct 31 19:45:03.569755 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:03.569637 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd" event={"ID":"934b6146-1f28-42ae-8d56-46e6dfcae403","Type":"ContainerDied","Data":"2e115922703e402b76626395fb1d7b3fb04caa372f65ecd0f651c7099734a5e4"} Oct 31 19:45:03.570460 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:03.570119 4910 scope.go:122] "RemoveContainer" containerID="2e115922703e402b76626395fb1d7b3fb04caa372f65ecd0f651c7099734a5e4" Oct 31 19:45:03.582578 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:03.582550 4910 generic.go:356] "Generic (PLEG): container finished" podID="b6e841b0-383b-41dc-9d35-05f57d92ea1d" containerID="5085c2a4d0b5e51421c95dbbd1dae226de6ecbc7a8149aca040f8780c7ce8b6b" exitCode=60 Oct 31 19:45:03.584000 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:03.583975 4910 scope.go:122] "RemoveContainer" containerID="5085c2a4d0b5e51421c95dbbd1dae226de6ecbc7a8149aca040f8780c7ce8b6b" Oct 31 19:45:03.585435 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:03.583974 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb" event={"ID":"b6e841b0-383b-41dc-9d35-05f57d92ea1d","Type":"ContainerDied","Data":"5085c2a4d0b5e51421c95dbbd1dae226de6ecbc7a8149aca040f8780c7ce8b6b"} Oct 31 19:45:03.595867 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:03.595288 4910 generic.go:356] "Generic (PLEG): container finished" podID="64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab" containerID="78534a7d6055a5e28116ac1d20724d2b615479e48e3a12873937732f4f4c9861" exitCode=60 Oct 31 19:45:03.595867 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:03.595853 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp" event={"ID":"64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab","Type":"ContainerDied","Data":"78534a7d6055a5e28116ac1d20724d2b615479e48e3a12873937732f4f4c9861"} Oct 31 19:45:03.598810 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:03.597696 4910 scope.go:122] "RemoveContainer" containerID="78534a7d6055a5e28116ac1d20724d2b615479e48e3a12873937732f4f4c9861" Oct 31 19:45:04.606491 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:04.606449 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd" event={"ID":"934b6146-1f28-42ae-8d56-46e6dfcae403","Type":"ContainerStarted","Data":"e8fdd28ddad5895e58de2101a312174122c12dcc97167415b1b766bb5f9be530"} Oct 31 19:45:04.609110 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:04.609075 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb" event={"ID":"b6e841b0-383b-41dc-9d35-05f57d92ea1d","Type":"ContainerStarted","Data":"54b136dabc3630f23c9c5e2d64233100ed646e4830ee405433430213c6092f4b"} Oct 31 19:45:04.611595 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:04.611565 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp" event={"ID":"64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab","Type":"ContainerStarted","Data":"c9463e82068b64ccc8bcc7d8041a2129e4c988e95cd4b67cb3091d4a8cfb74b2"} Oct 31 19:45:04.613976 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:04.613949 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq" event={"ID":"f1cfaf0d-c80e-4860-828d-7ea807d34ccc","Type":"ContainerStarted","Data":"f0c165bc0ad8bf34a4aef4db242da9a076d14b6c4418167782d26b35edc886bc"} Oct 31 19:45:04.615988 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:04.615964 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2" event={"ID":"c52c8a23-5d17-4b66-995e-beb28777c843","Type":"ContainerStarted","Data":"e42ca948e6f4e2b4e7a14680cb44c7b4ff85aebd1db13baf785cc5c24f52b957"} Oct 31 19:45:04.633506 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:04.633460 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd" podStartSLOduration=4.633447956 podStartE2EDuration="4.633447956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:04.625755057 +0000 UTC m=+1538.988575430" watchObservedRunningTime="2026-10-31 19:45:04.633447956 +0000 UTC m=+1538.996268328" Oct 31 19:45:04.648803 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:04.648758 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb" podStartSLOduration=4.648745926 podStartE2EDuration="4.648745926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:04.643426721 +0000 UTC m=+1539.006247096" watchObservedRunningTime="2026-10-31 19:45:04.648745926 +0000 UTC m=+1539.011566298" Oct 31 19:45:04.666248 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:04.666206 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp" podStartSLOduration=4.666193945 podStartE2EDuration="4.666193945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:04.661591492 +0000 UTC m=+1539.024411867" watchObservedRunningTime="2026-10-31 19:45:04.666193945 +0000 UTC m=+1539.029014317" Oct 31 19:45:04.685830 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:04.685776 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2" podStartSLOduration=4.685764014 podStartE2EDuration="4.685764014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:04.680693605 +0000 UTC m=+1539.043513977" watchObservedRunningTime="2026-10-31 19:45:04.685764014 +0000 UTC m=+1539.048584386" Oct 31 19:45:05.626094 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:05.626056 4910 generic.go:356] "Generic (PLEG): container finished" podID="f1cfaf0d-c80e-4860-828d-7ea807d34ccc" containerID="f0c165bc0ad8bf34a4aef4db242da9a076d14b6c4418167782d26b35edc886bc" exitCode=60 Oct 31 19:45:05.626572 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:05.626130 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq" event={"ID":"f1cfaf0d-c80e-4860-828d-7ea807d34ccc","Type":"ContainerDied","Data":"f0c165bc0ad8bf34a4aef4db242da9a076d14b6c4418167782d26b35edc886bc"} Oct 31 19:45:05.626572 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:05.626204 4910 scope.go:122] "RemoveContainer" containerID="8261ec38270042a62b232a0b9a6f7a98412a9da99f173df9e6ff035a2ee1eb0f" Oct 31 19:45:05.626572 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:05.626551 4910 scope.go:122] "RemoveContainer" containerID="f0c165bc0ad8bf34a4aef4db242da9a076d14b6c4418167782d26b35edc886bc" Oct 31 19:45:05.627014 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:45:05.626985 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq_odh-ai-gateway-infra(f1cfaf0d-c80e-4860-828d-7ea807d34ccc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq" podUID="f1cfaf0d-c80e-4860-828d-7ea807d34ccc" Oct 31 19:45:05.628776 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:05.628750 4910 generic.go:356] "Generic (PLEG): container finished" podID="c52c8a23-5d17-4b66-995e-beb28777c843" containerID="e42ca948e6f4e2b4e7a14680cb44c7b4ff85aebd1db13baf785cc5c24f52b957" exitCode=60 Oct 31 19:45:05.628942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:05.628839 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2" event={"ID":"c52c8a23-5d17-4b66-995e-beb28777c843","Type":"ContainerDied","Data":"e42ca948e6f4e2b4e7a14680cb44c7b4ff85aebd1db13baf785cc5c24f52b957"} Oct 31 19:45:05.629430 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:05.629235 4910 scope.go:122] "RemoveContainer" containerID="e42ca948e6f4e2b4e7a14680cb44c7b4ff85aebd1db13baf785cc5c24f52b957" Oct 31 19:45:05.629752 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:45:05.629705 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2_odh-ai-gateway-infra(c52c8a23-5d17-4b66-995e-beb28777c843)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2" podUID="c52c8a23-5d17-4b66-995e-beb28777c843" Oct 31 19:45:05.631379 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:05.631355 4910 generic.go:356] "Generic (PLEG): container finished" podID="934b6146-1f28-42ae-8d56-46e6dfcae403" containerID="e8fdd28ddad5895e58de2101a312174122c12dcc97167415b1b766bb5f9be530" exitCode=60 Oct 31 19:45:05.631481 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:05.631434 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd" event={"ID":"934b6146-1f28-42ae-8d56-46e6dfcae403","Type":"ContainerDied","Data":"e8fdd28ddad5895e58de2101a312174122c12dcc97167415b1b766bb5f9be530"} Oct 31 19:45:05.631742 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:05.631721 4910 scope.go:122] "RemoveContainer" containerID="e8fdd28ddad5895e58de2101a312174122c12dcc97167415b1b766bb5f9be530" Oct 31 19:45:05.632124 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:45:05.632098 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd_odh-ai-gateway-infra(934b6146-1f28-42ae-8d56-46e6dfcae403)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd" podUID="934b6146-1f28-42ae-8d56-46e6dfcae403" Oct 31 19:45:05.633759 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:05.633734 4910 generic.go:356] "Generic (PLEG): container finished" podID="b6e841b0-383b-41dc-9d35-05f57d92ea1d" containerID="54b136dabc3630f23c9c5e2d64233100ed646e4830ee405433430213c6092f4b" exitCode=60 Oct 31 19:45:05.633857 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:05.633811 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb" event={"ID":"b6e841b0-383b-41dc-9d35-05f57d92ea1d","Type":"ContainerDied","Data":"54b136dabc3630f23c9c5e2d64233100ed646e4830ee405433430213c6092f4b"} Oct 31 19:45:05.634084 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:05.634064 4910 scope.go:122] "RemoveContainer" containerID="54b136dabc3630f23c9c5e2d64233100ed646e4830ee405433430213c6092f4b" Oct 31 19:45:05.634413 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:45:05.634375 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb_odh-ai-gateway-infra(b6e841b0-383b-41dc-9d35-05f57d92ea1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb" podUID="b6e841b0-383b-41dc-9d35-05f57d92ea1d" Oct 31 19:45:05.635883 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:05.635864 4910 generic.go:356] "Generic (PLEG): container finished" podID="64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab" containerID="c9463e82068b64ccc8bcc7d8041a2129e4c988e95cd4b67cb3091d4a8cfb74b2" exitCode=60 Oct 31 19:45:05.636006 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:05.635986 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp" event={"ID":"64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab","Type":"ContainerDied","Data":"c9463e82068b64ccc8bcc7d8041a2129e4c988e95cd4b67cb3091d4a8cfb74b2"} Oct 31 19:45:05.636161 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:05.636148 4910 scope.go:122] "RemoveContainer" containerID="c9463e82068b64ccc8bcc7d8041a2129e4c988e95cd4b67cb3091d4a8cfb74b2" Oct 31 19:45:05.636412 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:45:05.636395 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp_odh-ai-gateway-infra(64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp" podUID="64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab" Oct 31 19:45:05.643483 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:05.643466 4910 scope.go:122] "RemoveContainer" containerID="66de7e52102798889f25ef0d6721a1398fac32f61b1fa5c62c01bb08956ea802" Oct 31 19:45:05.675450 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:05.675399 4910 scope.go:122] "RemoveContainer" containerID="2e115922703e402b76626395fb1d7b3fb04caa372f65ecd0f651c7099734a5e4" Oct 31 19:45:05.691224 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:05.691201 4910 scope.go:122] "RemoveContainer" containerID="5085c2a4d0b5e51421c95dbbd1dae226de6ecbc7a8149aca040f8780c7ce8b6b" Oct 31 19:45:05.708484 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:05.708445 4910 scope.go:122] "RemoveContainer" containerID="78534a7d6055a5e28116ac1d20724d2b615479e48e3a12873937732f4f4c9861" Oct 31 19:45:06.650774 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:06.649334 4910 scope.go:122] "RemoveContainer" containerID="f0c165bc0ad8bf34a4aef4db242da9a076d14b6c4418167782d26b35edc886bc" Oct 31 19:45:06.650774 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:45:06.649831 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq_odh-ai-gateway-infra(f1cfaf0d-c80e-4860-828d-7ea807d34ccc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq" podUID="f1cfaf0d-c80e-4860-828d-7ea807d34ccc" Oct 31 19:45:06.651390 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:06.651368 4910 scope.go:122] "RemoveContainer" containerID="e42ca948e6f4e2b4e7a14680cb44c7b4ff85aebd1db13baf785cc5c24f52b957" Oct 31 19:45:06.651795 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:45:06.651766 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2_odh-ai-gateway-infra(c52c8a23-5d17-4b66-995e-beb28777c843)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2" podUID="c52c8a23-5d17-4b66-995e-beb28777c843" Oct 31 19:45:06.653452 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:06.653431 4910 scope.go:122] "RemoveContainer" containerID="e8fdd28ddad5895e58de2101a312174122c12dcc97167415b1b766bb5f9be530" Oct 31 19:45:06.653846 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:45:06.653821 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd_odh-ai-gateway-infra(934b6146-1f28-42ae-8d56-46e6dfcae403)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd" podUID="934b6146-1f28-42ae-8d56-46e6dfcae403" Oct 31 19:45:06.655588 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:06.655562 4910 scope.go:122] "RemoveContainer" containerID="54b136dabc3630f23c9c5e2d64233100ed646e4830ee405433430213c6092f4b" Oct 31 19:45:06.656656 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:45:06.655872 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb_odh-ai-gateway-infra(b6e841b0-383b-41dc-9d35-05f57d92ea1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb" podUID="b6e841b0-383b-41dc-9d35-05f57d92ea1d" Oct 31 19:45:06.658073 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:06.658053 4910 scope.go:122] "RemoveContainer" containerID="c9463e82068b64ccc8bcc7d8041a2129e4c988e95cd4b67cb3091d4a8cfb74b2" Oct 31 19:45:06.658412 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:45:06.658388 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp_odh-ai-gateway-infra(64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp" podUID="64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab" Oct 31 19:45:09.957731 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:09.957696 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q"] Oct 31 19:45:09.993428 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:09.993391 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a813e2-6kw2q"] Oct 31 19:45:11.760089 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:11.760052 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bb47a2d-9025-4800-bf5d-85e8a881c4f1" path="/var/lib/kubelet/pods/7bb47a2d-9025-4800-bf5d-85e8a881c4f1/volumes" Oct 31 19:45:12.723581 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:45:12.723542 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:45:15.757883 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:15.757851 4910 scope.go:122] "RemoveContainer" containerID="e42ca948e6f4e2b4e7a14680cb44c7b4ff85aebd1db13baf785cc5c24f52b957" Oct 31 19:45:16.819704 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:16.819670 4910 generic.go:356] "Generic (PLEG): container finished" podID="c52c8a23-5d17-4b66-995e-beb28777c843" containerID="6a3429bb73e93d2830f0885e5c9fbe177b74c58d940c19ef4c2130b958825d75" exitCode=60 Oct 31 19:45:16.820247 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:16.819744 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2" event={"ID":"c52c8a23-5d17-4b66-995e-beb28777c843","Type":"ContainerDied","Data":"6a3429bb73e93d2830f0885e5c9fbe177b74c58d940c19ef4c2130b958825d75"} Oct 31 19:45:16.820247 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:16.819808 4910 scope.go:122] "RemoveContainer" containerID="e42ca948e6f4e2b4e7a14680cb44c7b4ff85aebd1db13baf785cc5c24f52b957" Oct 31 19:45:16.820373 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:16.820287 4910 scope.go:122] "RemoveContainer" containerID="6a3429bb73e93d2830f0885e5c9fbe177b74c58d940c19ef4c2130b958825d75" Oct 31 19:45:16.820691 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:45:16.820656 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2_odh-ai-gateway-infra(c52c8a23-5d17-4b66-995e-beb28777c843)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2" podUID="c52c8a23-5d17-4b66-995e-beb28777c843" Oct 31 19:45:17.748009 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:17.747967 4910 scope.go:122] "RemoveContainer" containerID="54b136dabc3630f23c9c5e2d64233100ed646e4830ee405433430213c6092f4b" Oct 31 19:45:17.853664 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:17.853631 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2"] Oct 31 19:45:18.130550 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:18.130519 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2" Oct 31 19:45:18.232613 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:18.232196 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c52c8a23-5d17-4b66-995e-beb28777c843-kube-api-access-k78xh\" (UniqueName: \"kubernetes.io/projected/c52c8a23-5d17-4b66-995e-beb28777c843-kube-api-access-k78xh\") pod \"c52c8a23-5d17-4b66-995e-beb28777c843\" (UID: \"c52c8a23-5d17-4b66-995e-beb28777c843\") " Oct 31 19:45:18.232613 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:18.232408 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c52c8a23-5d17-4b66-995e-beb28777c843-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c52c8a23-5d17-4b66-995e-beb28777c843-maas-api-service-ca\") pod \"c52c8a23-5d17-4b66-995e-beb28777c843\" (UID: \"c52c8a23-5d17-4b66-995e-beb28777c843\") " Oct 31 19:45:18.240586 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:18.233989 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52c8a23-5d17-4b66-995e-beb28777c843-maas-api-service-ca" pod "c52c8a23-5d17-4b66-995e-beb28777c843" (UID: "c52c8a23-5d17-4b66-995e-beb28777c843"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:45:18.308052 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:18.292167 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52c8a23-5d17-4b66-995e-beb28777c843-kube-api-access-k78xh" pod "c52c8a23-5d17-4b66-995e-beb28777c843" (UID: "c52c8a23-5d17-4b66-995e-beb28777c843"). InnerVolumeSpecName "kube-api-access-k78xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:45:18.343547 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:18.343508 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c52c8a23-5d17-4b66-995e-beb28777c843-maas-api-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:45:18.345273 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:18.345245 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k78xh\" (UniqueName: \"kubernetes.io/projected/c52c8a23-5d17-4b66-995e-beb28777c843-kube-api-access-k78xh\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:45:18.747891 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:18.747800 4910 scope.go:122] "RemoveContainer" containerID="c9463e82068b64ccc8bcc7d8041a2129e4c988e95cd4b67cb3091d4a8cfb74b2" Oct 31 19:45:18.848313 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:18.848276 4910 generic.go:356] "Generic (PLEG): container finished" podID="b6e841b0-383b-41dc-9d35-05f57d92ea1d" containerID="cf9f178f66d18325a09bb65d1f7ce375f9f613cfe9fde236bda237a558201f97" exitCode=60 Oct 31 19:45:18.848505 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:18.848349 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb" event={"ID":"b6e841b0-383b-41dc-9d35-05f57d92ea1d","Type":"ContainerDied","Data":"cf9f178f66d18325a09bb65d1f7ce375f9f613cfe9fde236bda237a558201f97"} Oct 31 19:45:18.848505 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:18.848424 4910 scope.go:122] "RemoveContainer" containerID="54b136dabc3630f23c9c5e2d64233100ed646e4830ee405433430213c6092f4b" Oct 31 19:45:18.848999 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:18.848978 4910 scope.go:122] "RemoveContainer" containerID="cf9f178f66d18325a09bb65d1f7ce375f9f613cfe9fde236bda237a558201f97" Oct 31 19:45:18.849530 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:45:18.849483 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb_odh-ai-gateway-infra(b6e841b0-383b-41dc-9d35-05f57d92ea1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb" podUID="b6e841b0-383b-41dc-9d35-05f57d92ea1d" Oct 31 19:45:18.850373 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:18.850341 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2" event={"ID":"c52c8a23-5d17-4b66-995e-beb28777c843","Type":"ContainerDied","Data":"dbf5fb25c9bc5a00b366787eea51c2a021ae791ce28a6e3dc1b79c3846f97e31"} Oct 31 19:45:18.850494 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:18.850347 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2" Oct 31 19:45:18.986697 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:18.986564 4910 scope.go:122] "RemoveContainer" containerID="6a3429bb73e93d2830f0885e5c9fbe177b74c58d940c19ef4c2130b958825d75" Oct 31 19:45:19.002499 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:19.002461 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2"] Oct 31 19:45:19.009467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:19.009437 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891265-vqjf2"] Oct 31 19:45:19.747447 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:19.747423 4910 scope.go:122] "RemoveContainer" containerID="f0c165bc0ad8bf34a4aef4db242da9a076d14b6c4418167782d26b35edc886bc" Oct 31 19:45:19.747623 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:19.747597 4910 scope.go:122] "RemoveContainer" containerID="e8fdd28ddad5895e58de2101a312174122c12dcc97167415b1b766bb5f9be530" Oct 31 19:45:19.766030 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:19.765997 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c52c8a23-5d17-4b66-995e-beb28777c843" path="/var/lib/kubelet/pods/c52c8a23-5d17-4b66-995e-beb28777c843/volumes" Oct 31 19:45:19.881168 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:19.880755 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp" event={"ID":"64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab","Type":"ContainerStarted","Data":"a07742a91f436841cc42e7135d7195ce8d92f0dc75723e4de62f5d074f4ebe03"} Oct 31 19:45:19.893403 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:19.890475 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb"] Oct 31 19:45:20.844865 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:20.844837 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb" Oct 31 19:45:20.898130 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:20.898075 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq" event={"ID":"f1cfaf0d-c80e-4860-828d-7ea807d34ccc","Type":"ContainerStarted","Data":"4e3948a233bf44f88fd8246165afbae6089a040787560a327d7c8862a31ab439"} Oct 31 19:45:20.901507 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:20.901472 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd" event={"ID":"934b6146-1f28-42ae-8d56-46e6dfcae403","Type":"ContainerStarted","Data":"ec0f8e5e06057840c4d7a84786b99f83aaef7493acd7a67720d40302eb41a338"} Oct 31 19:45:20.908568 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:20.908534 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb" event={"ID":"b6e841b0-383b-41dc-9d35-05f57d92ea1d","Type":"ContainerDied","Data":"3572ccec29e225c165e5aaa946ea5703be94e14f23c92d13849264595fd0d41b"} Oct 31 19:45:20.908745 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:20.908602 4910 scope.go:122] "RemoveContainer" containerID="cf9f178f66d18325a09bb65d1f7ce375f9f613cfe9fde236bda237a558201f97" Oct 31 19:45:20.908745 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:20.908693 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb" Oct 31 19:45:20.913240 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:20.913209 4910 generic.go:356] "Generic (PLEG): container finished" podID="64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab" containerID="a07742a91f436841cc42e7135d7195ce8d92f0dc75723e4de62f5d074f4ebe03" exitCode=60 Oct 31 19:45:20.913417 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:20.913393 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp" event={"ID":"64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab","Type":"ContainerDied","Data":"a07742a91f436841cc42e7135d7195ce8d92f0dc75723e4de62f5d074f4ebe03"} Oct 31 19:45:20.913965 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:20.913885 4910 scope.go:122] "RemoveContainer" containerID="a07742a91f436841cc42e7135d7195ce8d92f0dc75723e4de62f5d074f4ebe03" Oct 31 19:45:20.914408 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:45:20.914376 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp_odh-ai-gateway-infra(64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp" podUID="64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab" Oct 31 19:45:20.928869 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:20.928836 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6e841b0-383b-41dc-9d35-05f57d92ea1d-kube-api-access-4lhzt\" (UniqueName: \"kubernetes.io/projected/b6e841b0-383b-41dc-9d35-05f57d92ea1d-kube-api-access-4lhzt\") pod \"b6e841b0-383b-41dc-9d35-05f57d92ea1d\" (UID: \"b6e841b0-383b-41dc-9d35-05f57d92ea1d\") " Oct 31 19:45:20.929039 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:20.929029 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6e841b0-383b-41dc-9d35-05f57d92ea1d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6e841b0-383b-41dc-9d35-05f57d92ea1d-maas-api-service-ca\") pod \"b6e841b0-383b-41dc-9d35-05f57d92ea1d\" (UID: \"b6e841b0-383b-41dc-9d35-05f57d92ea1d\") " Oct 31 19:45:20.931175 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:20.931082 4910 scope.go:122] "RemoveContainer" containerID="c9463e82068b64ccc8bcc7d8041a2129e4c988e95cd4b67cb3091d4a8cfb74b2" Oct 31 19:45:20.931579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:20.931484 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6e841b0-383b-41dc-9d35-05f57d92ea1d-maas-api-service-ca" pod "b6e841b0-383b-41dc-9d35-05f57d92ea1d" (UID: "b6e841b0-383b-41dc-9d35-05f57d92ea1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:45:20.932310 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:20.932283 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e841b0-383b-41dc-9d35-05f57d92ea1d-kube-api-access-4lhzt" pod "b6e841b0-383b-41dc-9d35-05f57d92ea1d" (UID: "b6e841b0-383b-41dc-9d35-05f57d92ea1d"). InnerVolumeSpecName "kube-api-access-4lhzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:45:20.938244 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:20.938215 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp"] Oct 31 19:45:21.032359 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:21.031984 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6e841b0-383b-41dc-9d35-05f57d92ea1d-maas-api-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:45:21.032631 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:21.032604 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lhzt\" (UniqueName: \"kubernetes.io/projected/b6e841b0-383b-41dc-9d35-05f57d92ea1d-kube-api-access-4lhzt\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:45:21.268013 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:21.267978 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb"] Oct 31 19:45:21.273946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:21.273906 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891265-krwxb"] Oct 31 19:45:22.053077 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.053036 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6e841b0-383b-41dc-9d35-05f57d92ea1d" path="/var/lib/kubelet/pods/b6e841b0-383b-41dc-9d35-05f57d92ea1d/volumes" Oct 31 19:45:22.147143 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.147107 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq"] Oct 31 19:45:22.184938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.183523 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd"] Oct 31 19:45:22.197927 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.197869 4910 generic.go:356] "Generic (PLEG): container finished" podID="934b6146-1f28-42ae-8d56-46e6dfcae403" containerID="ec0f8e5e06057840c4d7a84786b99f83aaef7493acd7a67720d40302eb41a338" exitCode=60 Oct 31 19:45:22.199968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.198486 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd" event={"ID":"934b6146-1f28-42ae-8d56-46e6dfcae403","Type":"ContainerDied","Data":"ec0f8e5e06057840c4d7a84786b99f83aaef7493acd7a67720d40302eb41a338"} Oct 31 19:45:22.199968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.198573 4910 scope.go:122] "RemoveContainer" containerID="e8fdd28ddad5895e58de2101a312174122c12dcc97167415b1b766bb5f9be530" Oct 31 19:45:22.213254 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.213220 4910 generic.go:356] "Generic (PLEG): container finished" podID="f1cfaf0d-c80e-4860-828d-7ea807d34ccc" containerID="4e3948a233bf44f88fd8246165afbae6089a040787560a327d7c8862a31ab439" exitCode=60 Oct 31 19:45:22.213379 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.213337 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq" event={"ID":"f1cfaf0d-c80e-4860-828d-7ea807d34ccc","Type":"ContainerDied","Data":"4e3948a233bf44f88fd8246165afbae6089a040787560a327d7c8862a31ab439"} Oct 31 19:45:22.233897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.233870 4910 scope.go:122] "RemoveContainer" containerID="f0c165bc0ad8bf34a4aef4db242da9a076d14b6c4418167782d26b35edc886bc" Oct 31 19:45:22.583628 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.582852 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq" Oct 31 19:45:22.630692 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.630664 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd" Oct 31 19:45:22.646552 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.646511 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp" Oct 31 19:45:22.710686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.710651 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1cfaf0d-c80e-4860-828d-7ea807d34ccc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1cfaf0d-c80e-4860-828d-7ea807d34ccc-maas-api-service-ca\") pod \"f1cfaf0d-c80e-4860-828d-7ea807d34ccc\" (UID: \"f1cfaf0d-c80e-4860-828d-7ea807d34ccc\") " Oct 31 19:45:22.710873 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.710768 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1cfaf0d-c80e-4860-828d-7ea807d34ccc-kube-api-access-5mvx2\" (UniqueName: \"kubernetes.io/projected/f1cfaf0d-c80e-4860-828d-7ea807d34ccc-kube-api-access-5mvx2\") pod \"f1cfaf0d-c80e-4860-828d-7ea807d34ccc\" (UID: \"f1cfaf0d-c80e-4860-828d-7ea807d34ccc\") " Oct 31 19:45:22.711496 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.711442 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1cfaf0d-c80e-4860-828d-7ea807d34ccc-maas-api-service-ca" pod "f1cfaf0d-c80e-4860-828d-7ea807d34ccc" (UID: "f1cfaf0d-c80e-4860-828d-7ea807d34ccc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:45:22.711634 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.711613 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1cfaf0d-c80e-4860-828d-7ea807d34ccc-maas-api-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:45:22.748175 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.745471 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1cfaf0d-c80e-4860-828d-7ea807d34ccc-kube-api-access-5mvx2" pod "f1cfaf0d-c80e-4860-828d-7ea807d34ccc" (UID: "f1cfaf0d-c80e-4860-828d-7ea807d34ccc"). InnerVolumeSpecName "kube-api-access-5mvx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:45:22.813797 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.813733 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/934b6146-1f28-42ae-8d56-46e6dfcae403-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/934b6146-1f28-42ae-8d56-46e6dfcae403-maas-api-service-ca\") pod \"934b6146-1f28-42ae-8d56-46e6dfcae403\" (UID: \"934b6146-1f28-42ae-8d56-46e6dfcae403\") " Oct 31 19:45:22.820750 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.813896 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/934b6146-1f28-42ae-8d56-46e6dfcae403-kube-api-access-xm44v\" (UniqueName: \"kubernetes.io/projected/934b6146-1f28-42ae-8d56-46e6dfcae403-kube-api-access-xm44v\") pod \"934b6146-1f28-42ae-8d56-46e6dfcae403\" (UID: \"934b6146-1f28-42ae-8d56-46e6dfcae403\") " Oct 31 19:45:22.820958 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.820886 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab-kube-api-access-l7hvb\" (UniqueName: \"kubernetes.io/projected/64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab-kube-api-access-l7hvb\") pod \"64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab\" (UID: \"64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab\") " Oct 31 19:45:22.821061 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.819752 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/934b6146-1f28-42ae-8d56-46e6dfcae403-maas-api-service-ca" pod "934b6146-1f28-42ae-8d56-46e6dfcae403" (UID: "934b6146-1f28-42ae-8d56-46e6dfcae403"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:45:22.821196 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.821175 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab-maas-api-service-ca\") pod \"64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab\" (UID: \"64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab\") " Oct 31 19:45:22.822375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.822342 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mvx2\" (UniqueName: \"kubernetes.io/projected/f1cfaf0d-c80e-4860-828d-7ea807d34ccc-kube-api-access-5mvx2\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:45:22.822375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.822375 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/934b6146-1f28-42ae-8d56-46e6dfcae403-maas-api-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:45:22.823143 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.823119 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab-maas-api-service-ca" pod "64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab" (UID: "64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:45:22.824290 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.824253 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/934b6146-1f28-42ae-8d56-46e6dfcae403-kube-api-access-xm44v" pod "934b6146-1f28-42ae-8d56-46e6dfcae403" (UID: "934b6146-1f28-42ae-8d56-46e6dfcae403"). InnerVolumeSpecName "kube-api-access-xm44v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:45:22.826383 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.826334 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab-kube-api-access-l7hvb" pod "64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab" (UID: "64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab"). InnerVolumeSpecName "kube-api-access-l7hvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:45:22.925084 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.924983 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm44v\" (UniqueName: \"kubernetes.io/projected/934b6146-1f28-42ae-8d56-46e6dfcae403-kube-api-access-xm44v\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:45:22.925084 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.925018 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7hvb\" (UniqueName: \"kubernetes.io/projected/64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab-kube-api-access-l7hvb\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:45:22.925084 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:22.925035 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab-maas-api-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:45:23.225550 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:23.225463 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd" Oct 31 19:45:23.225550 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:23.225502 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd" event={"ID":"934b6146-1f28-42ae-8d56-46e6dfcae403","Type":"ContainerDied","Data":"79e9c40d0ff5afe05edce6e8054f9880ce5205ad7921ffc7daff88304a7247aa"} Oct 31 19:45:23.226151 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:23.225566 4910 scope.go:122] "RemoveContainer" containerID="ec0f8e5e06057840c4d7a84786b99f83aaef7493acd7a67720d40302eb41a338" Oct 31 19:45:23.234756 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:23.234649 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp" Oct 31 19:45:23.236492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:23.234641 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp" event={"ID":"64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab","Type":"ContainerDied","Data":"48ec2f808cb68439c6a277737f0f89ec614aca98f880699e9e6dea9365ffe122"} Oct 31 19:45:23.242825 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:23.242806 4910 scope.go:122] "RemoveContainer" containerID="a07742a91f436841cc42e7135d7195ce8d92f0dc75723e4de62f5d074f4ebe03" Oct 31 19:45:23.242988 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:23.242968 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq" event={"ID":"f1cfaf0d-c80e-4860-828d-7ea807d34ccc","Type":"ContainerDied","Data":"3c1f1026352cb0d68b2a32bf8542618e5fe1facb0436dcc4fe828090794f3c3b"} Oct 31 19:45:23.243077 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:23.243058 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq" Oct 31 19:45:23.257798 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:23.257770 4910 scope.go:122] "RemoveContainer" containerID="4e3948a233bf44f88fd8246165afbae6089a040787560a327d7c8862a31ab439" Oct 31 19:45:23.264366 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:23.264315 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd"] Oct 31 19:45:23.271287 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:23.271250 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891265-mhmsd"] Oct 31 19:45:23.281814 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:23.281792 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp"] Oct 31 19:45:23.286864 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:23.286843 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891265-k7jhp"] Oct 31 19:45:23.300927 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:23.298960 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq"] Oct 31 19:45:23.303179 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:23.303159 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891265-kw6mq"] Oct 31 19:45:23.762839 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:23.762760 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab" path="/var/lib/kubelet/pods/64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab/volumes" Oct 31 19:45:23.764366 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:23.764326 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="934b6146-1f28-42ae-8d56-46e6dfcae403" path="/var/lib/kubelet/pods/934b6146-1f28-42ae-8d56-46e6dfcae403/volumes" Oct 31 19:45:23.764953 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:23.764929 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1cfaf0d-c80e-4860-828d-7ea807d34ccc" path="/var/lib/kubelet/pods/f1cfaf0d-c80e-4860-828d-7ea807d34ccc/volumes" Oct 31 19:45:29.843846 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:29.843805 4910 scope.go:122] "RemoveContainer" containerID="a53cdf1855d4041730daa3e0be768c2d68a4e8b761e18cf3666c31766c1506cf" Oct 31 19:45:29.857477 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:29.857455 4910 scope.go:122] "RemoveContainer" containerID="1cbe3247361f3b913c24c22b33982f5689e62d63a00b4ace7a7179086f84ec33" Oct 31 19:45:29.868327 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:29.868300 4910 scope.go:122] "RemoveContainer" containerID="a8f55653b7c19e7a81939f9fdf790f36e4d31a8579515a9222f684c28f4d205b" Oct 31 19:45:35.511089 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:35.511011 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:45:35.637058 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:35.637015 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 19:45:35.746938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:35.744997 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:45:38.610327 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:38.610232 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:45:42.718557 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:45:42.718460 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:45:44.640037 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:44.639557 4910 patch_prober.go:29] interesting pod/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:45:44.640037 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:44.639681 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" podUID="c86e67dd-5de9-4213-b26d-fcbc3bd90353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:45:47.039391 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:47.039304 4910 generic.go:356] "Generic (PLEG): container finished" podID="6da8e70c-0b96-48e1-92cb-01ca01434b98" containerID="33a0f003beabc7d5156e32eb35fff9331b705e3c20a2f984b0a1ee0e9d068749" exitCode=0 Oct 31 19:45:47.039391 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:47.039369 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" event={"ID":"6da8e70c-0b96-48e1-92cb-01ca01434b98","Type":"ContainerDied","Data":"33a0f003beabc7d5156e32eb35fff9331b705e3c20a2f984b0a1ee0e9d068749"} Oct 31 19:45:47.039878 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:47.039830 4910 scope.go:122] "RemoveContainer" containerID="33a0f003beabc7d5156e32eb35fff9331b705e3c20a2f984b0a1ee0e9d068749" Oct 31 19:45:47.046498 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:47.046437 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:45:47.046720 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:47.046603 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:45:47.046720 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:47.046695 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:45:48.057858 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:48.057826 4910 generic.go:356] "Generic (PLEG): container finished" podID="c86e67dd-5de9-4213-b26d-fcbc3bd90353" containerID="61e8372de968812a78948b9889ca8ea08d70b5f79321f7283cf56b354d18f00a" exitCode=0 Oct 31 19:45:48.058399 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:48.057899 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" event={"ID":"c86e67dd-5de9-4213-b26d-fcbc3bd90353","Type":"ContainerDied","Data":"61e8372de968812a78948b9889ca8ea08d70b5f79321f7283cf56b354d18f00a"} Oct 31 19:45:48.058579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:48.058561 4910 scope.go:122] "RemoveContainer" containerID="61e8372de968812a78948b9889ca8ea08d70b5f79321f7283cf56b354d18f00a" Oct 31 19:45:48.060695 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:48.060672 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" event={"ID":"6da8e70c-0b96-48e1-92cb-01ca01434b98","Type":"ContainerStarted","Data":"eb9890bd5fc2b594175fc4dd1dfd6dfda00142fd531d62457d189ad341193e1b"} Oct 31 19:45:48.061284 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:48.061246 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:45:48.061380 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:48.061346 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:45:48.061442 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:48.061399 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:45:48.672420 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:48.672377 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:45:48.673613 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:48.673579 4910 patch_prober.go:29] interesting pod/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 19:45:48.673758 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:48.673643 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" podUID="6da8e70c-0b96-48e1-92cb-01ca01434b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 19:45:49.094599 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:49.094567 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" event={"ID":"c86e67dd-5de9-4213-b26d-fcbc3bd90353","Type":"ContainerStarted","Data":"418d0eab141ae1e42b12d7018a9cbf0f5f2f5afaa4c005214239ea3eedec345e"} Oct 31 19:45:49.676783 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:49.676739 4910 patch_prober.go:29] interesting pod/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 19:45:49.677055 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:49.676821 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" podUID="6da8e70c-0b96-48e1-92cb-01ca01434b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 19:45:49.708388 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:49.708347 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:45:49.708585 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:49.708540 4910 patch_prober.go:29] interesting pod/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 31 19:45:49.708657 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:49.708591 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" podUID="c86e67dd-5de9-4213-b26d-fcbc3bd90353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 31 19:45:50.672938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:50.672452 4910 patch_prober.go:29] interesting pod/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 19:45:50.672938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:50.672518 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" podUID="6da8e70c-0b96-48e1-92cb-01ca01434b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 19:45:50.708942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:50.708892 4910 patch_prober.go:29] interesting pod/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 31 19:45:50.709208 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:50.709183 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" podUID="c86e67dd-5de9-4213-b26d-fcbc3bd90353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 31 19:45:51.680936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:51.680322 4910 patch_prober.go:29] interesting pod/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 19:45:51.680936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:51.680398 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" podUID="6da8e70c-0b96-48e1-92cb-01ca01434b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 19:45:51.708845 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:51.708811 4910 patch_prober.go:29] interesting pod/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 31 19:45:51.709039 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:51.708859 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" podUID="c86e67dd-5de9-4213-b26d-fcbc3bd90353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 31 19:45:52.673935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:52.673310 4910 patch_prober.go:29] interesting pod/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 19:45:52.673935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:52.673370 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" podUID="6da8e70c-0b96-48e1-92cb-01ca01434b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 19:45:52.710937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:52.710299 4910 patch_prober.go:29] interesting pod/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 31 19:45:52.710937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:52.710376 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" podUID="c86e67dd-5de9-4213-b26d-fcbc3bd90353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 31 19:45:53.672852 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:53.672816 4910 patch_prober.go:29] interesting pod/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 19:45:53.673069 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:53.672868 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" podUID="6da8e70c-0b96-48e1-92cb-01ca01434b98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 19:45:53.709083 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:53.709043 4910 patch_prober.go:29] interesting pod/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 31 19:45:53.709278 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:53.709107 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" podUID="c86e67dd-5de9-4213-b26d-fcbc3bd90353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 31 19:45:54.714738 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:54.710069 4910 patch_prober.go:29] interesting pod/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body= Oct 31 19:45:54.714738 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:54.710149 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" podUID="c86e67dd-5de9-4213-b26d-fcbc3bd90353" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" Oct 31 19:45:54.719017 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:54.718973 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:45:54.720320 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:54.720294 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:45:54.744935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:54.734563 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:45:55.714972 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:55.714925 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:45:55.715353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:55.715256 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:45:55.716108 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:45:55.716092 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:46:12.716172 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:46:12.716126 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:46:26.975855 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:26.975816 4910 generic.go:356] "Generic (PLEG): container finished" podID="27663054-5fbb-4a52-b101-682623a5f518" containerID="cd393767ffcf2b3f30129e3923bb28f97ad899ff30836b6c05c36f1dd0d7f536" exitCode=0 Oct 31 19:46:26.976597 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:26.976020 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" event={"ID":"27663054-5fbb-4a52-b101-682623a5f518","Type":"ContainerDied","Data":"cd393767ffcf2b3f30129e3923bb28f97ad899ff30836b6c05c36f1dd0d7f536"} Oct 31 19:46:26.977224 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:26.977196 4910 scope.go:122] "RemoveContainer" containerID="cd393767ffcf2b3f30129e3923bb28f97ad899ff30836b6c05c36f1dd0d7f536" Oct 31 19:46:26.985770 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:26.985723 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:46:26.985894 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:26.985831 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:46:26.985894 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:26.985882 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:46:27.173658 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:27.173622 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:46:27.996458 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:27.996419 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" event={"ID":"27663054-5fbb-4a52-b101-682623a5f518","Type":"ContainerStarted","Data":"ff6908f3775a8a99746ba2197dd51d1ab4ddac11d84b04ad757ec521ca4a3fde"} Oct 31 19:46:28.102276 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:28.102236 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:46:28.103477 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:28.103446 4910 patch_prober.go:29] interesting pod/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 31 19:46:28.103635 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:28.103509 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" podUID="27663054-5fbb-4a52-b101-682623a5f518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 31 19:46:29.101544 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:29.101506 4910 patch_prober.go:29] interesting pod/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 31 19:46:29.102003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:29.101577 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" podUID="27663054-5fbb-4a52-b101-682623a5f518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 31 19:46:29.963938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:29.959134 4910 scope.go:122] "RemoveContainer" containerID="41e7fb032dface71aa0ba3931c99c036f6a79fe5f6a11832789bc46e8200faf4" Oct 31 19:46:29.982854 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:29.982826 4910 scope.go:122] "RemoveContainer" containerID="b8723fa6b341308336772e9d198c21065e4ed08d29c591901b5f09a1434da676" Oct 31 19:46:29.994382 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:29.994360 4910 scope.go:122] "RemoveContainer" containerID="a9b25acd432e61b4abcd5337f6e9d9957192dc3687df8c22a65517af0e656ea8" Oct 31 19:46:30.102021 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:30.101992 4910 patch_prober.go:29] interesting pod/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 31 19:46:30.102464 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:30.102435 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" podUID="27663054-5fbb-4a52-b101-682623a5f518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 31 19:46:31.102233 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:31.102199 4910 patch_prober.go:29] interesting pod/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 31 19:46:31.102602 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:31.102252 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" podUID="27663054-5fbb-4a52-b101-682623a5f518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 31 19:46:32.102096 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:32.102058 4910 patch_prober.go:29] interesting pod/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 31 19:46:32.102279 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:32.102128 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" podUID="27663054-5fbb-4a52-b101-682623a5f518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 31 19:46:33.101950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:33.101902 4910 patch_prober.go:29] interesting pod/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 31 19:46:33.102155 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:33.101970 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" podUID="27663054-5fbb-4a52-b101-682623a5f518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 31 19:46:34.110967 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:34.110930 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:46:34.111491 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:34.111439 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:46:34.112292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:46:34.112271 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:46:42.714904 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:46:42.714862 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:47:06.774814 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:06.774781 4910 generic.go:356] "Generic (PLEG): container finished" podID="7631819a-de14-4a11-a5a9-9b2e05e7c7a6" containerID="de17dab6e975433bab5a8de52bfc85dda1d4fa6131db47c3d80ed5afce1ab072" exitCode=0 Oct 31 19:47:06.775312 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:06.774818 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" event={"ID":"7631819a-de14-4a11-a5a9-9b2e05e7c7a6","Type":"ContainerDied","Data":"de17dab6e975433bab5a8de52bfc85dda1d4fa6131db47c3d80ed5afce1ab072"} Oct 31 19:47:06.775312 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:06.775259 4910 scope.go:122] "RemoveContainer" containerID="de17dab6e975433bab5a8de52bfc85dda1d4fa6131db47c3d80ed5afce1ab072" Oct 31 19:47:06.777478 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:06.777440 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:47:06.777593 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:06.777534 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:47:06.777659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:06.777589 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:47:07.799750 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:07.799719 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" event={"ID":"7631819a-de14-4a11-a5a9-9b2e05e7c7a6","Type":"ContainerStarted","Data":"f67913e4013c7eead267489dbcc8a1556ee654c25b9ebefa9d6bb63dd6592bd0"} Oct 31 19:47:08.255676 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:08.255647 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:47:08.256989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:08.256955 4910 patch_prober.go:29] interesting pod/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 31 19:47:08.257230 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:08.257202 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" podUID="7631819a-de14-4a11-a5a9-9b2e05e7c7a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 31 19:47:09.255637 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:09.255593 4910 patch_prober.go:29] interesting pod/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 31 19:47:09.256079 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:09.255658 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" podUID="7631819a-de14-4a11-a5a9-9b2e05e7c7a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 31 19:47:10.037109 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:10.037073 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e86eb251-mbhb4"] Oct 31 19:47:10.063305 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:10.063270 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e86eb251-mbhb4"] Oct 31 19:47:10.255613 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:10.255576 4910 patch_prober.go:29] interesting pod/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 31 19:47:10.255821 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:10.255641 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" podUID="7631819a-de14-4a11-a5a9-9b2e05e7c7a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 31 19:47:10.779146 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:10.779112 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:47:11.255116 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:11.255076 4910 patch_prober.go:29] interesting pod/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 31 19:47:11.255307 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:11.255143 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" podUID="7631819a-de14-4a11-a5a9-9b2e05e7c7a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 31 19:47:11.765862 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:11.765821 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29ded46-17c9-4189-8401-99218bbbf54a" path="/var/lib/kubelet/pods/a29ded46-17c9-4189-8401-99218bbbf54a/volumes" Oct 31 19:47:12.255643 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:12.255603 4910 patch_prober.go:29] interesting pod/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 31 19:47:12.255844 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:12.255668 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" podUID="7631819a-de14-4a11-a5a9-9b2e05e7c7a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 31 19:47:12.715336 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:47:12.715295 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:47:13.255797 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:13.255750 4910 patch_prober.go:29] interesting pod/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 31 19:47:13.256355 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:13.255806 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" podUID="7631819a-de14-4a11-a5a9-9b2e05e7c7a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 31 19:47:14.290848 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:14.290805 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:47:14.296580 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:14.296486 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:47:30.071106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:47:30.071071 4910 scope.go:122] "RemoveContainer" containerID="97ca3b39271a1a9e89a91d6b3ee4fd054359438202062afe357e000d266128f8" Oct 31 19:47:42.715158 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:47:42.715114 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:48:00.845097 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:00.845057 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt"] Oct 31 19:48:00.846063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:00.846025 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" podUID="69515b0c-d32a-40b2-baf8-995554f2a5b9" containerName="istio-proxy" containerID="cri-o://8ec59dc90c2c484fed199e1a4a9204e47f52ebddd09cc8d7a7378d99759d04ae" gracePeriod=30 Oct 31 19:48:05.472713 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:05.472674 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n"] Oct 31 19:48:05.473531 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:05.473494 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" podUID="32a18536-b3f0-42cc-a148-cc5fc0c24dca" containerName="istio-proxy" containerID="cri-o://e5ed391d101671489df9e96e4e053de9eb6110506a483aec9bfeec4f2b97a097" gracePeriod=30 Oct 31 19:48:05.594134 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:05.594096 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl"] Oct 31 19:48:05.594677 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:05.594639 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" podUID="2e946922-c762-4b9d-93af-73ab4af9fd50" containerName="istio-proxy" containerID="cri-o://01dd692f3d43a76e19c9fc79a14e16ffeee83aad69aa782fc081e5f776c7eb3d" gracePeriod=30 Oct 31 19:48:06.186634 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.186605 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:48:06.225333 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.225295 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk"] Oct 31 19:48:06.226129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226106 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c52c8a23-5d17-4b66-995e-beb28777c843" containerName="cleanup" Oct 31 19:48:06.226129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226125 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c52c8a23-5d17-4b66-995e-beb28777c843" containerName="cleanup" Oct 31 19:48:06.226129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226137 4910 cpu_manager.go:414] "containerMap: removing container" podUID="64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab" containerName="cleanup" Oct 31 19:48:06.226375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226142 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab" containerName="cleanup" Oct 31 19:48:06.226375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226155 4910 cpu_manager.go:414] "containerMap: removing container" podUID="934b6146-1f28-42ae-8d56-46e6dfcae403" containerName="cleanup" Oct 31 19:48:06.226375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226160 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="934b6146-1f28-42ae-8d56-46e6dfcae403" containerName="cleanup" Oct 31 19:48:06.226375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226168 4910 cpu_manager.go:414] "containerMap: removing container" podUID="b6e841b0-383b-41dc-9d35-05f57d92ea1d" containerName="cleanup" Oct 31 19:48:06.226375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226172 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6e841b0-383b-41dc-9d35-05f57d92ea1d" containerName="cleanup" Oct 31 19:48:06.226375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226188 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c52c8a23-5d17-4b66-995e-beb28777c843" containerName="cleanup" Oct 31 19:48:06.226375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226192 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c52c8a23-5d17-4b66-995e-beb28777c843" containerName="cleanup" Oct 31 19:48:06.226375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226200 4910 cpu_manager.go:414] "containerMap: removing container" podUID="b6e841b0-383b-41dc-9d35-05f57d92ea1d" containerName="cleanup" Oct 31 19:48:06.226375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226205 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6e841b0-383b-41dc-9d35-05f57d92ea1d" containerName="cleanup" Oct 31 19:48:06.226375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226214 4910 cpu_manager.go:414] "containerMap: removing container" podUID="934b6146-1f28-42ae-8d56-46e6dfcae403" containerName="cleanup" Oct 31 19:48:06.226375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226218 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="934b6146-1f28-42ae-8d56-46e6dfcae403" containerName="cleanup" Oct 31 19:48:06.226375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226233 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c52c8a23-5d17-4b66-995e-beb28777c843" containerName="cleanup" Oct 31 19:48:06.226375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226239 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c52c8a23-5d17-4b66-995e-beb28777c843" containerName="cleanup" Oct 31 19:48:06.226375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226253 4910 cpu_manager.go:414] "containerMap: removing container" podUID="934b6146-1f28-42ae-8d56-46e6dfcae403" containerName="cleanup" Oct 31 19:48:06.226375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226259 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="934b6146-1f28-42ae-8d56-46e6dfcae403" containerName="cleanup" Oct 31 19:48:06.226375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226270 4910 cpu_manager.go:414] "containerMap: removing container" podUID="f1cfaf0d-c80e-4860-828d-7ea807d34ccc" containerName="cleanup" Oct 31 19:48:06.226375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226279 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1cfaf0d-c80e-4860-828d-7ea807d34ccc" containerName="cleanup" Oct 31 19:48:06.226375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226295 4910 cpu_manager.go:414] "containerMap: removing container" podUID="64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab" containerName="cleanup" Oct 31 19:48:06.226375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226304 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab" containerName="cleanup" Oct 31 19:48:06.226375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226316 4910 cpu_manager.go:414] "containerMap: removing container" podUID="f1cfaf0d-c80e-4860-828d-7ea807d34ccc" containerName="cleanup" Oct 31 19:48:06.226375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226322 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1cfaf0d-c80e-4860-828d-7ea807d34ccc" containerName="cleanup" Oct 31 19:48:06.226375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226336 4910 cpu_manager.go:414] "containerMap: removing container" podUID="69515b0c-d32a-40b2-baf8-995554f2a5b9" containerName="istio-proxy" Oct 31 19:48:06.226375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226343 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69515b0c-d32a-40b2-baf8-995554f2a5b9" containerName="istio-proxy" Oct 31 19:48:06.227537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226535 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1cfaf0d-c80e-4860-828d-7ea807d34ccc" containerName="cleanup" Oct 31 19:48:06.227537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226548 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab" containerName="cleanup" Oct 31 19:48:06.227537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226556 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6e841b0-383b-41dc-9d35-05f57d92ea1d" containerName="cleanup" Oct 31 19:48:06.227537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226563 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab" containerName="cleanup" Oct 31 19:48:06.227537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226571 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1cfaf0d-c80e-4860-828d-7ea807d34ccc" containerName="cleanup" Oct 31 19:48:06.227537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226579 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1cfaf0d-c80e-4860-828d-7ea807d34ccc" containerName="cleanup" Oct 31 19:48:06.227537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226591 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="69515b0c-d32a-40b2-baf8-995554f2a5b9" containerName="istio-proxy" Oct 31 19:48:06.227537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226604 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c52c8a23-5d17-4b66-995e-beb28777c843" containerName="cleanup" Oct 31 19:48:06.227537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226612 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="934b6146-1f28-42ae-8d56-46e6dfcae403" containerName="cleanup" Oct 31 19:48:06.227537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226620 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6e841b0-383b-41dc-9d35-05f57d92ea1d" containerName="cleanup" Oct 31 19:48:06.227537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226630 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c52c8a23-5d17-4b66-995e-beb28777c843" containerName="cleanup" Oct 31 19:48:06.227537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226638 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6e841b0-383b-41dc-9d35-05f57d92ea1d" containerName="cleanup" Oct 31 19:48:06.227537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226647 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="934b6146-1f28-42ae-8d56-46e6dfcae403" containerName="cleanup" Oct 31 19:48:06.227537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226658 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c52c8a23-5d17-4b66-995e-beb28777c843" containerName="cleanup" Oct 31 19:48:06.227537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.226665 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="934b6146-1f28-42ae-8d56-46e6dfcae403" containerName="cleanup" Oct 31 19:48:06.230671 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.230647 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.252528 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.252482 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk"] Oct 31 19:48:06.332391 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.332274 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69515b0c-d32a-40b2-baf8-995554f2a5b9-kube-api-access-rkrfn\" (UniqueName: \"kubernetes.io/projected/69515b0c-d32a-40b2-baf8-995554f2a5b9-kube-api-access-rkrfn\") pod \"69515b0c-d32a-40b2-baf8-995554f2a5b9\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " Oct 31 19:48:06.334681 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.334636 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-workload-socket\") pod \"69515b0c-d32a-40b2-baf8-995554f2a5b9\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " Oct 31 19:48:06.335118 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.335084 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-workload-socket" pod "69515b0c-d32a-40b2-baf8-995554f2a5b9" (UID: "69515b0c-d32a-40b2-baf8-995554f2a5b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:06.335260 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.335171 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-workload-certs\") pod \"69515b0c-d32a-40b2-baf8-995554f2a5b9\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " Oct 31 19:48:06.335260 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.335222 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-token\" (UniqueName: \"kubernetes.io/projected/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-token\") pod \"69515b0c-d32a-40b2-baf8-995554f2a5b9\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " Oct 31 19:48:06.335388 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.335273 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69515b0c-d32a-40b2-baf8-995554f2a5b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69515b0c-d32a-40b2-baf8-995554f2a5b9-istiod-ca-cert\") pod \"69515b0c-d32a-40b2-baf8-995554f2a5b9\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " Oct 31 19:48:06.335388 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.335348 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-data\") pod \"69515b0c-d32a-40b2-baf8-995554f2a5b9\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " Oct 31 19:48:06.335550 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.335529 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-podinfo\") pod \"69515b0c-d32a-40b2-baf8-995554f2a5b9\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " Oct 31 19:48:06.335716 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.335698 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-envoy\") pod \"69515b0c-d32a-40b2-baf8-995554f2a5b9\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " Oct 31 19:48:06.335779 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.335745 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-credential-socket\") pod \"69515b0c-d32a-40b2-baf8-995554f2a5b9\" (UID: \"69515b0c-d32a-40b2-baf8-995554f2a5b9\") " Oct 31 19:48:06.337414 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.336484 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-workload-certs" pod "69515b0c-d32a-40b2-baf8-995554f2a5b9" (UID: "69515b0c-d32a-40b2-baf8-995554f2a5b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:06.337414 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.336720 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-credential-socket" pod "69515b0c-d32a-40b2-baf8-995554f2a5b9" (UID: "69515b0c-d32a-40b2-baf8-995554f2a5b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:06.337414 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.336794 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69515b0c-d32a-40b2-baf8-995554f2a5b9-istiod-ca-cert" pod "69515b0c-d32a-40b2-baf8-995554f2a5b9" (UID: "69515b0c-d32a-40b2-baf8-995554f2a5b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:48:06.337414 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.337318 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-data" pod "69515b0c-d32a-40b2-baf8-995554f2a5b9" (UID: "69515b0c-d32a-40b2-baf8-995554f2a5b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:06.340775 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.340743 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-workload-certs\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.340929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.340863 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-credential-socket\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.341011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.340993 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-envoy\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.341073 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.341043 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-data\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.341140 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.341088 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bx9m\" (UniqueName: \"kubernetes.io/projected/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-kube-api-access-6bx9m\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.341219 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.341204 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-podinfo\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.341324 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.341310 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-token\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.341399 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.341384 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istiod-ca-cert\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.341459 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.341415 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-workload-socket\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.341583 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.341568 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:06.341993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.341905 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:06.342098 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.342079 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69515b0c-d32a-40b2-baf8-995554f2a5b9-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:06.342163 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.342099 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:06.342163 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.342114 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:06.344438 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.344412 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-podinfo" pod "69515b0c-d32a-40b2-baf8-995554f2a5b9" (UID: "69515b0c-d32a-40b2-baf8-995554f2a5b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:48:06.344738 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.344701 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-envoy" pod "69515b0c-d32a-40b2-baf8-995554f2a5b9" (UID: "69515b0c-d32a-40b2-baf8-995554f2a5b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:06.346277 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.346255 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69515b0c-d32a-40b2-baf8-995554f2a5b9-kube-api-access-rkrfn" pod "69515b0c-d32a-40b2-baf8-995554f2a5b9" (UID: "69515b0c-d32a-40b2-baf8-995554f2a5b9"). InnerVolumeSpecName "kube-api-access-rkrfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:48:06.347483 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.347463 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-token" pod "69515b0c-d32a-40b2-baf8-995554f2a5b9" (UID: "69515b0c-d32a-40b2-baf8-995554f2a5b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:48:06.444260 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.444152 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-envoy\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.444260 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.444228 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-data\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.444517 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.444273 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bx9m\" (UniqueName: \"kubernetes.io/projected/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-kube-api-access-6bx9m\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.444517 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.444364 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-podinfo\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.444517 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.444457 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-token\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.444741 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.444522 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istiod-ca-cert\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.444741 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.444550 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-workload-socket\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.444741 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.444686 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-data\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.444900 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.444777 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-workload-certs\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.444900 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.444865 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-credential-socket\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.445149 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.445022 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:06.445149 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.445079 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-workload-socket\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.445149 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.445086 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkrfn\" (UniqueName: \"kubernetes.io/projected/69515b0c-d32a-40b2-baf8-995554f2a5b9-kube-api-access-rkrfn\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:06.445149 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.445137 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:06.445399 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.445154 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69515b0c-d32a-40b2-baf8-995554f2a5b9-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:06.445399 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.445155 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-workload-certs\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.445399 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.445309 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-credential-socket\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.445498 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.445457 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istiod-ca-cert\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.446750 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.446727 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-envoy\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.447049 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.447032 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-podinfo\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.456891 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.456861 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-token\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.457107 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.457087 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bx9m\" (UniqueName: \"kubernetes.io/projected/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-kube-api-access-6bx9m\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.510660 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.510621 4910 generic.go:356] "Generic (PLEG): container finished" podID="69515b0c-d32a-40b2-baf8-995554f2a5b9" containerID="8ec59dc90c2c484fed199e1a4a9204e47f52ebddd09cc8d7a7378d99759d04ae" exitCode=0 Oct 31 19:48:06.511166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.510672 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" Oct 31 19:48:06.511166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.510695 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" event={"ID":"69515b0c-d32a-40b2-baf8-995554f2a5b9","Type":"ContainerDied","Data":"8ec59dc90c2c484fed199e1a4a9204e47f52ebddd09cc8d7a7378d99759d04ae"} Oct 31 19:48:06.511166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.510751 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt" event={"ID":"69515b0c-d32a-40b2-baf8-995554f2a5b9","Type":"ContainerDied","Data":"ed6620e044e13c01bd0f20d8fd3e9d2629d5f0eadda07beac6392beefbce3b82"} Oct 31 19:48:06.511166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.510775 4910 scope.go:122] "RemoveContainer" containerID="8ec59dc90c2c484fed199e1a4a9204e47f52ebddd09cc8d7a7378d99759d04ae" Oct 31 19:48:06.523202 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.523180 4910 scope.go:122] "RemoveContainer" containerID="8ec59dc90c2c484fed199e1a4a9204e47f52ebddd09cc8d7a7378d99759d04ae" Oct 31 19:48:06.523521 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:48:06.523501 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ec59dc90c2c484fed199e1a4a9204e47f52ebddd09cc8d7a7378d99759d04ae\": container with ID starting with 8ec59dc90c2c484fed199e1a4a9204e47f52ebddd09cc8d7a7378d99759d04ae not found: ID does not exist" containerID="8ec59dc90c2c484fed199e1a4a9204e47f52ebddd09cc8d7a7378d99759d04ae" Oct 31 19:48:06.523602 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.523535 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ec59dc90c2c484fed199e1a4a9204e47f52ebddd09cc8d7a7378d99759d04ae"} err="failed to get container status \"8ec59dc90c2c484fed199e1a4a9204e47f52ebddd09cc8d7a7378d99759d04ae\": rpc error: code = NotFound desc = could not find container \"8ec59dc90c2c484fed199e1a4a9204e47f52ebddd09cc8d7a7378d99759d04ae\": container with ID starting with 8ec59dc90c2c484fed199e1a4a9204e47f52ebddd09cc8d7a7378d99759d04ae not found: ID does not exist" Oct 31 19:48:06.543078 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.543047 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt"] Oct 31 19:48:06.550401 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.550371 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-htmwt"] Oct 31 19:48:06.551183 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.551164 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:06.900729 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:48:06.900699 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b3d8c4_ff06_44a2_a9af_d945b5d238ae.slice/crio-83b185de930718276d8a6cf5d6c9b76d2e8d58eb5d72d484b7a9acba33ed88de WatchSource:0}: Error finding container 83b185de930718276d8a6cf5d6c9b76d2e8d58eb5d72d484b7a9acba33ed88de: Status 404 returned error can't find the container with id 83b185de930718276d8a6cf5d6c9b76d2e8d58eb5d72d484b7a9acba33ed88de Oct 31 19:48:06.901693 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.901653 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk"] Oct 31 19:48:06.904256 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.904098 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:48:06.904256 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.904184 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:48:06.904256 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:06.904236 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:48:07.521901 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:07.521856 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" event={"ID":"67b3d8c4-ff06-44a2-a9af-d945b5d238ae","Type":"ContainerStarted","Data":"eef4eefc3606e9183c784521515adbd1b503e105642ae42b0f2d12ed5ea0cee1"} Oct 31 19:48:07.522408 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:07.521973 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" event={"ID":"67b3d8c4-ff06-44a2-a9af-d945b5d238ae","Type":"ContainerStarted","Data":"83b185de930718276d8a6cf5d6c9b76d2e8d58eb5d72d484b7a9acba33ed88de"} Oct 31 19:48:07.551373 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:07.551338 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:07.553026 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:07.552996 4910 patch_prober.go:29] interesting pod/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 31 19:48:07.553158 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:07.553055 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" podUID="67b3d8c4-ff06-44a2-a9af-d945b5d238ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 31 19:48:07.557383 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:07.557330 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" podStartSLOduration=90.557312707 podStartE2EDuration="1m30.557312707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:07.548821523 +0000 UTC m=+1721.911641896" watchObservedRunningTime="2026-10-31 19:48:07.557312707 +0000 UTC m=+1721.920133081" Oct 31 19:48:07.769101 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:07.769065 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69515b0c-d32a-40b2-baf8-995554f2a5b9" path="/var/lib/kubelet/pods/69515b0c-d32a-40b2-baf8-995554f2a5b9/volumes" Oct 31 19:48:08.554933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:08.552483 4910 patch_prober.go:29] interesting pod/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 31 19:48:08.554933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:08.552556 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" podUID="67b3d8c4-ff06-44a2-a9af-d945b5d238ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 31 19:48:09.553309 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:09.553274 4910 patch_prober.go:29] interesting pod/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 31 19:48:09.553562 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:09.553529 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" podUID="67b3d8c4-ff06-44a2-a9af-d945b5d238ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 31 19:48:10.552593 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.552549 4910 patch_prober.go:29] interesting pod/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 31 19:48:10.553180 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.552628 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" podUID="67b3d8c4-ff06-44a2-a9af-d945b5d238ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 31 19:48:10.663588 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.663549 4910 generic.go:356] "Generic (PLEG): container finished" podID="32a18536-b3f0-42cc-a148-cc5fc0c24dca" containerID="e5ed391d101671489df9e96e4e053de9eb6110506a483aec9bfeec4f2b97a097" exitCode=0 Oct 31 19:48:10.663865 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.663844 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" event={"ID":"32a18536-b3f0-42cc-a148-cc5fc0c24dca","Type":"ContainerDied","Data":"e5ed391d101671489df9e96e4e053de9eb6110506a483aec9bfeec4f2b97a097"} Oct 31 19:48:10.894783 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.894754 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:48:10.973950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.972535 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9"] Oct 31 19:48:10.973950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.973639 4910 cpu_manager.go:414] "containerMap: removing container" podUID="64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab" containerName="cleanup" Oct 31 19:48:10.973950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.973661 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab" containerName="cleanup" Oct 31 19:48:10.973950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.973680 4910 cpu_manager.go:414] "containerMap: removing container" podUID="b6e841b0-383b-41dc-9d35-05f57d92ea1d" containerName="cleanup" Oct 31 19:48:10.973950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.973687 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6e841b0-383b-41dc-9d35-05f57d92ea1d" containerName="cleanup" Oct 31 19:48:10.973950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.973744 4910 cpu_manager.go:414] "containerMap: removing container" podUID="32a18536-b3f0-42cc-a148-cc5fc0c24dca" containerName="istio-proxy" Oct 31 19:48:10.973950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.973752 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32a18536-b3f0-42cc-a148-cc5fc0c24dca" containerName="istio-proxy" Oct 31 19:48:10.973950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.973773 4910 cpu_manager.go:414] "containerMap: removing container" podUID="f1cfaf0d-c80e-4860-828d-7ea807d34ccc" containerName="cleanup" Oct 31 19:48:10.973950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.973781 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1cfaf0d-c80e-4860-828d-7ea807d34ccc" containerName="cleanup" Oct 31 19:48:10.976941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.974671 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="32a18536-b3f0-42cc-a148-cc5fc0c24dca" containerName="istio-proxy" Oct 31 19:48:10.976941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.974701 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ca2d1b-b5a6-4516-8046-d3f8d4fd4dab" containerName="cleanup" Oct 31 19:48:10.979955 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.979584 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32a18536-b3f0-42cc-a148-cc5fc0c24dca-kube-api-access-ljz9q\" (UniqueName: \"kubernetes.io/projected/32a18536-b3f0-42cc-a148-cc5fc0c24dca-kube-api-access-ljz9q\") pod \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " Oct 31 19:48:10.979955 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.979641 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-workload-socket\") pod \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " Oct 31 19:48:10.979955 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.979755 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-workload-certs\") pod \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " Oct 31 19:48:10.979955 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.979821 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-podinfo\") pod \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " Oct 31 19:48:10.979955 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.979857 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-data\") pod \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " Oct 31 19:48:10.979955 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.979890 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-token\" (UniqueName: \"kubernetes.io/projected/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-token\") pod \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " Oct 31 19:48:10.979955 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.979940 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-envoy\") pod \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " Oct 31 19:48:10.980393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.980013 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-credential-socket\") pod \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " Oct 31 19:48:10.980393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.980157 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istiod-ca-cert\") pod \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\" (UID: \"32a18536-b3f0-42cc-a148-cc5fc0c24dca\") " Oct 31 19:48:10.981041 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.980963 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-credential-socket" pod "32a18536-b3f0-42cc-a148-cc5fc0c24dca" (UID: "32a18536-b3f0-42cc-a148-cc5fc0c24dca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:10.981202 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.981179 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-workload-certs" pod "32a18536-b3f0-42cc-a148-cc5fc0c24dca" (UID: "32a18536-b3f0-42cc-a148-cc5fc0c24dca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:10.981508 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.981395 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istiod-ca-cert" pod "32a18536-b3f0-42cc-a148-cc5fc0c24dca" (UID: "32a18536-b3f0-42cc-a148-cc5fc0c24dca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:48:10.981626 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.981474 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-workload-socket" pod "32a18536-b3f0-42cc-a148-cc5fc0c24dca" (UID: "32a18536-b3f0-42cc-a148-cc5fc0c24dca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:10.981730 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.981668 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:10.981983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.981959 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-data" pod "32a18536-b3f0-42cc-a148-cc5fc0c24dca" (UID: "32a18536-b3f0-42cc-a148-cc5fc0c24dca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:10.982756 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.982699 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9"] Oct 31 19:48:10.985413 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.985391 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-envoy" pod "32a18536-b3f0-42cc-a148-cc5fc0c24dca" (UID: "32a18536-b3f0-42cc-a148-cc5fc0c24dca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:10.985540 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.985488 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-token" pod "32a18536-b3f0-42cc-a148-cc5fc0c24dca" (UID: "32a18536-b3f0-42cc-a148-cc5fc0c24dca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:48:10.986728 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.986704 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7c892a-openshift-default-dockercfg-zdlhk\"" Oct 31 19:48:10.991510 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.989226 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-podinfo" pod "32a18536-b3f0-42cc-a148-cc5fc0c24dca" (UID: "32a18536-b3f0-42cc-a148-cc5fc0c24dca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:48:10.991510 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:10.990735 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a18536-b3f0-42cc-a148-cc5fc0c24dca-kube-api-access-ljz9q" pod "32a18536-b3f0-42cc-a148-cc5fc0c24dca" (UID: "32a18536-b3f0-42cc-a148-cc5fc0c24dca"). InnerVolumeSpecName "kube-api-access-ljz9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:48:11.082694 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.082610 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-credential-socket\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.082694 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.082653 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-workload-certs\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.082694 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.082670 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-token\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.082945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.082764 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-podinfo\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.082945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.082815 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-envoy\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.082945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.082930 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-workload-socket\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.083051 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.082959 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-data\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.083051 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.082988 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istiod-ca-cert\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.083120 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.083108 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj86c\" (UniqueName: \"kubernetes.io/projected/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-kube-api-access-vj86c\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.083240 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.083226 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:11.083276 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.083243 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:11.083276 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.083253 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljz9q\" (UniqueName: \"kubernetes.io/projected/32a18536-b3f0-42cc-a148-cc5fc0c24dca-kube-api-access-ljz9q\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:11.083276 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.083264 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:11.083276 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.083273 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:11.083403 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.083282 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:11.083403 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.083290 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:11.083403 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.083298 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:11.083403 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.083306 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32a18536-b3f0-42cc-a148-cc5fc0c24dca-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:11.187244 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.187189 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-workload-socket\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.187450 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.187251 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-data\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.187450 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.187307 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istiod-ca-cert\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.187450 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.187434 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj86c\" (UniqueName: \"kubernetes.io/projected/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-kube-api-access-vj86c\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.188148 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.187687 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-credential-socket\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.188148 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.187724 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-workload-certs\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.188148 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.187747 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-token\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.188148 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.187796 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-podinfo\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.188148 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.187999 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-envoy\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.193494 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.190328 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-data\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.193494 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.190561 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-workload-socket\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.193494 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.191370 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-workload-certs\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.193494 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.192032 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istiod-ca-cert\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.193863 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.193491 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-credential-socket\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.195218 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.195191 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-envoy\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.196268 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.196245 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-podinfo\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.206125 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.206101 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj86c\" (UniqueName: \"kubernetes.io/projected/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-kube-api-access-vj86c\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.206878 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.206858 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-token\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.366453 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.366419 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:11.566949 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.566621 4910 patch_prober.go:29] interesting pod/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 31 19:48:11.566949 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.566666 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" podUID="67b3d8c4-ff06-44a2-a9af-d945b5d238ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 31 19:48:11.674353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.674280 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" Oct 31 19:48:11.674353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.674332 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n" event={"ID":"32a18536-b3f0-42cc-a148-cc5fc0c24dca","Type":"ContainerDied","Data":"4c0fd0c511b8aa79c48bdc029c8cd434cda1be13cdbd8aae3ace2b706356b1fb"} Oct 31 19:48:11.674596 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.674386 4910 scope.go:122] "RemoveContainer" containerID="e5ed391d101671489df9e96e4e053de9eb6110506a483aec9bfeec4f2b97a097" Oct 31 19:48:11.675701 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.675662 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:48:11.677691 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.677655 4910 generic.go:356] "Generic (PLEG): container finished" podID="2e946922-c762-4b9d-93af-73ab4af9fd50" containerID="01dd692f3d43a76e19c9fc79a14e16ffeee83aad69aa782fc081e5f776c7eb3d" exitCode=0 Oct 31 19:48:11.677890 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.677866 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" event={"ID":"2e946922-c762-4b9d-93af-73ab4af9fd50","Type":"ContainerDied","Data":"01dd692f3d43a76e19c9fc79a14e16ffeee83aad69aa782fc081e5f776c7eb3d"} Oct 31 19:48:11.677995 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.677960 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" event={"ID":"2e946922-c762-4b9d-93af-73ab4af9fd50","Type":"ContainerDied","Data":"921a50b1f1216615809d4ab491a217117623203af1bd024d982a8265898fc63c"} Oct 31 19:48:11.690850 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.690830 4910 scope.go:122] "RemoveContainer" containerID="01dd692f3d43a76e19c9fc79a14e16ffeee83aad69aa782fc081e5f776c7eb3d" Oct 31 19:48:11.701034 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.701009 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-data\") pod \"2e946922-c762-4b9d-93af-73ab4af9fd50\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " Oct 31 19:48:11.701155 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.701057 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-envoy\") pod \"2e946922-c762-4b9d-93af-73ab4af9fd50\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " Oct 31 19:48:11.701155 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.701129 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-workload-certs\") pod \"2e946922-c762-4b9d-93af-73ab4af9fd50\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " Oct 31 19:48:11.701278 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.701166 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e946922-c762-4b9d-93af-73ab4af9fd50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e946922-c762-4b9d-93af-73ab4af9fd50-istiod-ca-cert\") pod \"2e946922-c762-4b9d-93af-73ab4af9fd50\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " Oct 31 19:48:11.701278 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.701259 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-token\" (UniqueName: \"kubernetes.io/projected/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-token\") pod \"2e946922-c762-4b9d-93af-73ab4af9fd50\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " Oct 31 19:48:11.701406 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.701391 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-workload-socket\") pod \"2e946922-c762-4b9d-93af-73ab4af9fd50\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " Oct 31 19:48:11.701473 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.701426 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e946922-c762-4b9d-93af-73ab4af9fd50-kube-api-access-bft7j\" (UniqueName: \"kubernetes.io/projected/2e946922-c762-4b9d-93af-73ab4af9fd50-kube-api-access-bft7j\") pod \"2e946922-c762-4b9d-93af-73ab4af9fd50\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " Oct 31 19:48:11.701473 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.701445 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-podinfo\") pod \"2e946922-c762-4b9d-93af-73ab4af9fd50\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " Oct 31 19:48:11.701590 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.701481 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-credential-socket\") pod \"2e946922-c762-4b9d-93af-73ab4af9fd50\" (UID: \"2e946922-c762-4b9d-93af-73ab4af9fd50\") " Oct 31 19:48:11.701590 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.701561 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e946922-c762-4b9d-93af-73ab4af9fd50-istiod-ca-cert" pod "2e946922-c762-4b9d-93af-73ab4af9fd50" (UID: "2e946922-c762-4b9d-93af-73ab4af9fd50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:48:11.701792 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.701764 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-workload-socket" pod "2e946922-c762-4b9d-93af-73ab4af9fd50" (UID: "2e946922-c762-4b9d-93af-73ab4af9fd50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:11.701864 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.701842 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-data" pod "2e946922-c762-4b9d-93af-73ab4af9fd50" (UID: "2e946922-c762-4b9d-93af-73ab4af9fd50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:11.702191 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.702170 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-workload-certs" pod "2e946922-c762-4b9d-93af-73ab4af9fd50" (UID: "2e946922-c762-4b9d-93af-73ab4af9fd50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:11.702434 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.702406 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:11.702520 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.702442 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:11.702520 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.702459 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:11.702520 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.702473 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e946922-c762-4b9d-93af-73ab4af9fd50-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:11.702690 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.702621 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-credential-socket" pod "2e946922-c762-4b9d-93af-73ab4af9fd50" (UID: "2e946922-c762-4b9d-93af-73ab4af9fd50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:11.704045 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.703862 4910 scope.go:122] "RemoveContainer" containerID="01dd692f3d43a76e19c9fc79a14e16ffeee83aad69aa782fc081e5f776c7eb3d" Oct 31 19:48:11.704761 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:48:11.704733 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01dd692f3d43a76e19c9fc79a14e16ffeee83aad69aa782fc081e5f776c7eb3d\": container with ID starting with 01dd692f3d43a76e19c9fc79a14e16ffeee83aad69aa782fc081e5f776c7eb3d not found: ID does not exist" containerID="01dd692f3d43a76e19c9fc79a14e16ffeee83aad69aa782fc081e5f776c7eb3d" Oct 31 19:48:11.704850 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.704775 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01dd692f3d43a76e19c9fc79a14e16ffeee83aad69aa782fc081e5f776c7eb3d"} err="failed to get container status \"01dd692f3d43a76e19c9fc79a14e16ffeee83aad69aa782fc081e5f776c7eb3d\": rpc error: code = NotFound desc = could not find container \"01dd692f3d43a76e19c9fc79a14e16ffeee83aad69aa782fc081e5f776c7eb3d\": container with ID starting with 01dd692f3d43a76e19c9fc79a14e16ffeee83aad69aa782fc081e5f776c7eb3d not found: ID does not exist" Oct 31 19:48:11.705293 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.705252 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e946922-c762-4b9d-93af-73ab4af9fd50-kube-api-access-bft7j" pod "2e946922-c762-4b9d-93af-73ab4af9fd50" (UID: "2e946922-c762-4b9d-93af-73ab4af9fd50"). InnerVolumeSpecName "kube-api-access-bft7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:48:11.706641 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.706605 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-token" pod "2e946922-c762-4b9d-93af-73ab4af9fd50" (UID: "2e946922-c762-4b9d-93af-73ab4af9fd50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:48:11.707420 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.707384 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-podinfo" pod "2e946922-c762-4b9d-93af-73ab4af9fd50" (UID: "2e946922-c762-4b9d-93af-73ab4af9fd50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:48:11.709375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.709343 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-envoy" pod "2e946922-c762-4b9d-93af-73ab4af9fd50" (UID: "2e946922-c762-4b9d-93af-73ab4af9fd50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:48:11.750789 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.749205 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx"] Oct 31 19:48:11.750789 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.750385 4910 cpu_manager.go:414] "containerMap: removing container" podUID="2e946922-c762-4b9d-93af-73ab4af9fd50" containerName="istio-proxy" Oct 31 19:48:11.750789 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.750407 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e946922-c762-4b9d-93af-73ab4af9fd50" containerName="istio-proxy" Oct 31 19:48:11.750789 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.750622 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e946922-c762-4b9d-93af-73ab4af9fd50" containerName="istio-proxy" Oct 31 19:48:11.776023 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.772111 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n"] Oct 31 19:48:11.776023 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.772238 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.805837 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.805765 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istiod-ca-cert\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.806059 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.805890 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-podinfo\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.806059 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.806006 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-credential-socket\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.806189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.806139 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-workload-certs\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.806246 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.806191 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-envoy\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.806307 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.806243 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h668j\" (UniqueName: \"kubernetes.io/projected/73aa5edd-70dd-4bf1-b51c-ba31774baacd-kube-api-access-h668j\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.806307 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.806282 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-data\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.806418 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.806342 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-token\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.806418 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.806366 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-workload-socket\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.807875 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.806518 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bft7j\" (UniqueName: \"kubernetes.io/projected/2e946922-c762-4b9d-93af-73ab4af9fd50-kube-api-access-bft7j\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:11.807875 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.806539 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:11.807875 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.806553 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:11.807875 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.806567 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:11.807875 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.806582 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e946922-c762-4b9d-93af-73ab4af9fd50-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:48:11.810892 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.810869 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-86j6n"] Oct 31 19:48:11.811256 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.811240 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx"] Oct 31 19:48:11.874276 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.848536 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9"] Oct 31 19:48:11.875698 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:48:11.875635 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c03ae04_2fbb_44b2_ac0d_25c39f35608b.slice/crio-119f5383195ab64f44711c7a6e3bd272c7a30df7e0a9582d5106bcc479f2fb88 WatchSource:0}: Error finding container 119f5383195ab64f44711c7a6e3bd272c7a30df7e0a9582d5106bcc479f2fb88: Status 404 returned error can't find the container with id 119f5383195ab64f44711c7a6e3bd272c7a30df7e0a9582d5106bcc479f2fb88 Oct 31 19:48:11.890226 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.890176 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:48:11.890352 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.890276 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:48:11.890352 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.890310 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:48:11.911951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.908438 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istiod-ca-cert\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.911951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.908480 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-podinfo\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.911951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.908514 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-credential-socket\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.911951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.908569 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-workload-certs\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.911951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.908596 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-envoy\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.911951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.908624 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h668j\" (UniqueName: \"kubernetes.io/projected/73aa5edd-70dd-4bf1-b51c-ba31774baacd-kube-api-access-h668j\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.911951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.908645 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-data\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.911951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.908677 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-token\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.911951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.908695 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-workload-socket\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.911951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.909068 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-workload-socket\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.911951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.909832 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-data\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.911951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.909891 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istiod-ca-cert\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.913199 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.910348 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-credential-socket\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.913199 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.911188 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-workload-certs\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.913199 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.911445 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-envoy\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.914179 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.914161 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-podinfo\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.956164 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.956076 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h668j\" (UniqueName: \"kubernetes.io/projected/73aa5edd-70dd-4bf1-b51c-ba31774baacd-kube-api-access-h668j\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:11.956349 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:11.956189 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-token\") pod \"e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:12.113875 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:12.113580 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:12.494378 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:12.494351 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx"] Oct 31 19:48:12.500385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:12.500248 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:48:12.500501 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:12.500343 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:48:12.500567 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:12.500524 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:48:12.554891 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:12.554855 4910 patch_prober.go:29] interesting pod/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 31 19:48:12.555027 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:12.554979 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" podUID="67b3d8c4-ff06-44a2-a9af-d945b5d238ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 31 19:48:12.660319 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:12.660198 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:48:12.660902 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:12.660857 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" podUID="2e946922-c762-4b9d-93af-73ab4af9fd50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:48:12.692034 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:12.692001 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" event={"ID":"73aa5edd-70dd-4bf1-b51c-ba31774baacd","Type":"ContainerStarted","Data":"829b63d1304303165e998b19b6f0533b2844e98a282b813938c9b4f0eb472463"} Oct 31 19:48:12.696060 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:12.696027 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" event={"ID":"7c03ae04-2fbb-44b2-ac0d-25c39f35608b","Type":"ContainerStarted","Data":"4af5eafc113132b8b8d8692f3602fae614f5c32cea538c31eee71d45e9dd83d6"} Oct 31 19:48:12.696188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:12.696103 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" event={"ID":"7c03ae04-2fbb-44b2-ac0d-25c39f35608b","Type":"ContainerStarted","Data":"119f5383195ab64f44711c7a6e3bd272c7a30df7e0a9582d5106bcc479f2fb88"} Oct 31 19:48:12.698273 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:12.698250 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl" Oct 31 19:48:12.778257 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:48:12.772841 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:48:12.809025 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:12.806818 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" podStartSLOduration=70.806799258 podStartE2EDuration="1m10.806799258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:12.768683513 +0000 UTC m=+1727.131503887" watchObservedRunningTime="2026-10-31 19:48:12.806799258 +0000 UTC m=+1727.169619636" Oct 31 19:48:12.809025 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:12.807731 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl"] Oct 31 19:48:12.815107 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:12.815073 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-lljdl"] Oct 31 19:48:13.368037 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:13.367978 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:13.378175 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:13.378148 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:13.552105 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:13.552070 4910 patch_prober.go:29] interesting pod/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Oct 31 19:48:13.552291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:13.552137 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" podUID="67b3d8c4-ff06-44a2-a9af-d945b5d238ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Oct 31 19:48:13.710319 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:13.710240 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" event={"ID":"73aa5edd-70dd-4bf1-b51c-ba31774baacd","Type":"ContainerStarted","Data":"4c4da181b1412575aa17cdee271cab49c42f6fffa7433b1400012c4ed753756c"} Oct 31 19:48:13.710834 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:13.710808 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:13.711729 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:13.711709 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:48:13.737117 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:13.737062 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" podStartSLOduration=54.737048167 podStartE2EDuration="54.737048167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:13.736141654 +0000 UTC m=+1728.098962027" watchObservedRunningTime="2026-10-31 19:48:13.737048167 +0000 UTC m=+1728.099868538" Oct 31 19:48:13.763506 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:13.763467 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e946922-c762-4b9d-93af-73ab4af9fd50" path="/var/lib/kubelet/pods/2e946922-c762-4b9d-93af-73ab4af9fd50/volumes" Oct 31 19:48:13.764335 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:13.764313 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32a18536-b3f0-42cc-a148-cc5fc0c24dca" path="/var/lib/kubelet/pods/32a18536-b3f0-42cc-a148-cc5fc0c24dca/volumes" Oct 31 19:48:14.114291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:14.114256 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:14.115687 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:14.115657 4910 patch_prober.go:29] interesting pod/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 31 19:48:14.115848 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:14.115720 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" podUID="73aa5edd-70dd-4bf1-b51c-ba31774baacd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 31 19:48:14.562275 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:14.562245 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:14.564007 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:14.563983 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:14.566189 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:14.566158 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:48:15.115032 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:15.114991 4910 patch_prober.go:29] interesting pod/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 31 19:48:15.115486 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:15.115061 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" podUID="73aa5edd-70dd-4bf1-b51c-ba31774baacd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 31 19:48:16.114734 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:16.114698 4910 patch_prober.go:29] interesting pod/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 31 19:48:16.114965 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:16.114757 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" podUID="73aa5edd-70dd-4bf1-b51c-ba31774baacd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 31 19:48:16.963871 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:16.963798 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv"] Oct 31 19:48:16.969461 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:16.969441 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv" Oct 31 19:48:16.974404 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:16.974375 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7c892a\"" Oct 31 19:48:16.975218 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:16.975197 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-7c892a-dockercfg-j2rrp\"" Oct 31 19:48:16.979965 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:16.979941 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv"] Oct 31 19:48:17.135937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:17.134123 4910 patch_prober.go:29] interesting pod/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 31 19:48:17.135937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:17.134202 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" podUID="73aa5edd-70dd-4bf1-b51c-ba31774baacd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 31 19:48:17.204932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:17.197593 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfz6r\" (UniqueName: \"kubernetes.io/projected/63a3c457-7089-49bf-8ed5-6b18b933b832-kube-api-access-cfz6r\") pod \"payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv\" (UID: \"63a3c457-7089-49bf-8ed5-6b18b933b832\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv" Oct 31 19:48:17.204932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:17.197711 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a3c457-7089-49bf-8ed5-6b18b933b832-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv\" (UID: \"63a3c457-7089-49bf-8ed5-6b18b933b832\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv" Oct 31 19:48:17.310983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:17.305166 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfz6r\" (UniqueName: \"kubernetes.io/projected/63a3c457-7089-49bf-8ed5-6b18b933b832-kube-api-access-cfz6r\") pod \"payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv\" (UID: \"63a3c457-7089-49bf-8ed5-6b18b933b832\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv" Oct 31 19:48:17.310983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:17.305298 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a3c457-7089-49bf-8ed5-6b18b933b832-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv\" (UID: \"63a3c457-7089-49bf-8ed5-6b18b933b832\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv" Oct 31 19:48:17.310983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:17.306679 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a3c457-7089-49bf-8ed5-6b18b933b832-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv\" (UID: \"63a3c457-7089-49bf-8ed5-6b18b933b832\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv" Oct 31 19:48:17.374935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:17.371612 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfz6r\" (UniqueName: \"kubernetes.io/projected/63a3c457-7089-49bf-8ed5-6b18b933b832-kube-api-access-cfz6r\") pod \"payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv\" (UID: \"63a3c457-7089-49bf-8ed5-6b18b933b832\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv" Oct 31 19:48:17.595986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:17.595302 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv" Oct 31 19:48:18.096938 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:48:18.089024 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a3c457_7089_49bf_8ed5_6b18b933b832.slice/crio-2ab27a890cc2f991e334ed10d5a0e8d89b354461ffb7f564ea4a555eacedf4bd WatchSource:0}: Error finding container 2ab27a890cc2f991e334ed10d5a0e8d89b354461ffb7f564ea4a555eacedf4bd: Status 404 returned error can't find the container with id 2ab27a890cc2f991e334ed10d5a0e8d89b354461ffb7f564ea4a555eacedf4bd Oct 31 19:48:18.100596 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:18.100563 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv"] Oct 31 19:48:18.115355 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:18.115312 4910 patch_prober.go:29] interesting pod/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 31 19:48:18.115482 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:18.115366 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" podUID="73aa5edd-70dd-4bf1-b51c-ba31774baacd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 31 19:48:18.859924 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:18.859865 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv" event={"ID":"63a3c457-7089-49bf-8ed5-6b18b933b832","Type":"ContainerStarted","Data":"bb817e5db3213ca1dbf220f8be639e9b103be94bd004d5f5013adedf7f3907c6"} Oct 31 19:48:18.860111 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:18.859950 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv" event={"ID":"63a3c457-7089-49bf-8ed5-6b18b933b832","Type":"ContainerStarted","Data":"2ab27a890cc2f991e334ed10d5a0e8d89b354461ffb7f564ea4a555eacedf4bd"} Oct 31 19:48:18.860111 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:18.859979 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv" Oct 31 19:48:18.883277 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:18.883224 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv" podStartSLOduration=2.883211508 podStartE2EDuration="2.883211508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:18.878862642 +0000 UTC m=+1733.241683014" watchObservedRunningTime="2026-10-31 19:48:18.883211508 +0000 UTC m=+1733.246031880" Oct 31 19:48:19.115345 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:19.115251 4910 patch_prober.go:29] interesting pod/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body= Oct 31 19:48:19.116035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:19.115999 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" podUID="73aa5edd-70dd-4bf1-b51c-ba31774baacd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" Oct 31 19:48:20.138964 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:20.138928 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:20.140106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:20.140072 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:20.143529 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:20.141939 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:48:29.959972 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:29.950508 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv" Oct 31 19:48:30.156777 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:30.156727 4910 scope.go:122] "RemoveContainer" containerID="f6b5f827976801d42108f1d68eacbf99da94673783e085b8ce71ccb546edae38" Oct 31 19:48:30.184639 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:48:30.184609 4910 scope.go:122] "RemoveContainer" containerID="694188f399057fff8517fe9525ad3d10c51ca5aac7137f6126c6dabc070dcc52" Oct 31 19:48:42.714509 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:48:42.714469 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:49:12.715639 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:49:12.715593 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:49:26.316398 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:26.316365 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 19:49:26.335492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:26.335465 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 19:49:30.252439 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:30.252402 4910 scope.go:122] "RemoveContainer" containerID="5cbfeabcd5fd57973c570c07874a694e25b026d82397df92014d39152de8dd67" Oct 31 19:49:30.264813 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:30.264790 4910 scope.go:122] "RemoveContainer" containerID="8115cc157caf6bb6cebc5eb34257bac13cf78f45e1727245bd3ed3fa789de338" Oct 31 19:49:40.140349 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:40.140311 4910 generic.go:356] "Generic (PLEG): container finished" podID="7c03ae04-2fbb-44b2-ac0d-25c39f35608b" containerID="4af5eafc113132b8b8d8692f3602fae614f5c32cea538c31eee71d45e9dd83d6" exitCode=0 Oct 31 19:49:40.140810 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:40.140354 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" event={"ID":"7c03ae04-2fbb-44b2-ac0d-25c39f35608b","Type":"ContainerDied","Data":"4af5eafc113132b8b8d8692f3602fae614f5c32cea538c31eee71d45e9dd83d6"} Oct 31 19:49:40.140810 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:40.140780 4910 scope.go:122] "RemoveContainer" containerID="4af5eafc113132b8b8d8692f3602fae614f5c32cea538c31eee71d45e9dd83d6" Oct 31 19:49:40.142823 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:40.142785 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:49:40.142998 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:40.142864 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:49:40.142998 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:40.142901 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:49:41.184070 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:41.183217 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" event={"ID":"7c03ae04-2fbb-44b2-ac0d-25c39f35608b","Type":"ContainerStarted","Data":"6f10b490f9dc5d9246a2c39544d0dd923911d9a5c5949bca71b21e918fbe8ef9"} Oct 31 19:49:41.367820 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:41.367786 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:49:41.368921 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:41.368878 4910 patch_prober.go:29] interesting pod/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 31 19:49:41.369052 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:41.368960 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" podUID="7c03ae04-2fbb-44b2-ac0d-25c39f35608b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 31 19:49:42.367330 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:42.367296 4910 patch_prober.go:29] interesting pod/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 31 19:49:42.367730 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:42.367383 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" podUID="7c03ae04-2fbb-44b2-ac0d-25c39f35608b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 31 19:49:42.723003 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:49:42.719173 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:49:43.367396 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:43.367360 4910 patch_prober.go:29] interesting pod/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 31 19:49:43.367801 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:43.367444 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" podUID="7c03ae04-2fbb-44b2-ac0d-25c39f35608b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 31 19:49:43.748020 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:43.745303 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:49:44.382077 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:44.382038 4910 patch_prober.go:29] interesting pod/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 31 19:49:44.382630 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:44.382116 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" podUID="7c03ae04-2fbb-44b2-ac0d-25c39f35608b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 31 19:49:45.367729 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:45.367688 4910 patch_prober.go:29] interesting pod/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 31 19:49:45.367957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:45.367765 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" podUID="7c03ae04-2fbb-44b2-ac0d-25c39f35608b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 31 19:49:46.368007 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:46.367961 4910 patch_prober.go:29] interesting pod/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Oct 31 19:49:46.368493 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:46.368034 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" podUID="7c03ae04-2fbb-44b2-ac0d-25c39f35608b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Oct 31 19:49:47.374694 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:47.374664 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:49:47.376187 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:49:47.376162 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:50:06.967713 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:06.967630 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5"] Oct 31 19:50:06.968555 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:06.967843 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" podUID="5775b1d1-5ece-4ebd-9582-2efad1157d76" containerName="istio-proxy" containerID="cri-o://1fcd0786951b5e2577d527f85a080de437a29c6a8aafba18acda0e5c08dbd5ed" gracePeriod=30 Oct 31 19:50:08.747145 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:08.747105 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65"] Oct 31 19:50:08.747665 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:08.747580 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" podUID="4ca75657-04c9-4283-9059-220e69657f7c" containerName="istio-proxy" containerID="cri-o://4a3cea06a4855de38ba869100c1ccd037ca61a071422c740548fcc7204a8f628" gracePeriod=30 Oct 31 19:50:12.789705 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:12.789671 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-gj6s5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 31 19:50:12.791386 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:12.791350 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" podUID="5775b1d1-5ece-4ebd-9582-2efad1157d76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 31 19:50:12.877956 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:12.866062 4910 patch_prober.go:29] interesting pod/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:50:12.877956 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:12.866172 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" podUID="4ca75657-04c9-4283-9059-220e69657f7c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:50:12.911966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:12.903945 4910 generic.go:356] "Generic (PLEG): container finished" podID="5775b1d1-5ece-4ebd-9582-2efad1157d76" containerID="1fcd0786951b5e2577d527f85a080de437a29c6a8aafba18acda0e5c08dbd5ed" exitCode=0 Oct 31 19:50:12.911966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:12.904268 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" event={"ID":"5775b1d1-5ece-4ebd-9582-2efad1157d76","Type":"ContainerDied","Data":"1fcd0786951b5e2577d527f85a080de437a29c6a8aafba18acda0e5c08dbd5ed"} Oct 31 19:50:12.911966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:12.907000 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:50:12.943268 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:50:12.941717 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:50:13.587460 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.587434 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:50:13.665770 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.665684 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-podinfo\") pod \"5775b1d1-5ece-4ebd-9582-2efad1157d76\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " Oct 31 19:50:13.665770 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.665749 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-credential-socket\") pod \"5775b1d1-5ece-4ebd-9582-2efad1157d76\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " Oct 31 19:50:13.666079 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.665782 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-workload-certs\") pod \"5775b1d1-5ece-4ebd-9582-2efad1157d76\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " Oct 31 19:50:13.666079 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.665809 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-data\") pod \"5775b1d1-5ece-4ebd-9582-2efad1157d76\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " Oct 31 19:50:13.666079 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.665849 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5775b1d1-5ece-4ebd-9582-2efad1157d76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5775b1d1-5ece-4ebd-9582-2efad1157d76-istiod-ca-cert\") pod \"5775b1d1-5ece-4ebd-9582-2efad1157d76\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " Oct 31 19:50:13.666079 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.666024 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-workload-socket\") pod \"5775b1d1-5ece-4ebd-9582-2efad1157d76\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " Oct 31 19:50:13.666079 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.666071 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-token\" (UniqueName: \"kubernetes.io/projected/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-token\") pod \"5775b1d1-5ece-4ebd-9582-2efad1157d76\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " Oct 31 19:50:13.666377 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.666145 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5775b1d1-5ece-4ebd-9582-2efad1157d76-kube-api-access-d2c54\" (UniqueName: \"kubernetes.io/projected/5775b1d1-5ece-4ebd-9582-2efad1157d76-kube-api-access-d2c54\") pod \"5775b1d1-5ece-4ebd-9582-2efad1157d76\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " Oct 31 19:50:13.666377 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.666159 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-credential-socket" pod "5775b1d1-5ece-4ebd-9582-2efad1157d76" (UID: "5775b1d1-5ece-4ebd-9582-2efad1157d76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:13.666377 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.666175 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-workload-certs" pod "5775b1d1-5ece-4ebd-9582-2efad1157d76" (UID: "5775b1d1-5ece-4ebd-9582-2efad1157d76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:13.666377 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.666292 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-envoy\") pod \"5775b1d1-5ece-4ebd-9582-2efad1157d76\" (UID: \"5775b1d1-5ece-4ebd-9582-2efad1157d76\") " Oct 31 19:50:13.666607 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.666388 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-workload-socket" pod "5775b1d1-5ece-4ebd-9582-2efad1157d76" (UID: "5775b1d1-5ece-4ebd-9582-2efad1157d76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:13.666607 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.666452 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5775b1d1-5ece-4ebd-9582-2efad1157d76-istiod-ca-cert" pod "5775b1d1-5ece-4ebd-9582-2efad1157d76" (UID: "5775b1d1-5ece-4ebd-9582-2efad1157d76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:50:13.666607 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.666591 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-data" pod "5775b1d1-5ece-4ebd-9582-2efad1157d76" (UID: "5775b1d1-5ece-4ebd-9582-2efad1157d76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:13.667243 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.667217 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5775b1d1-5ece-4ebd-9582-2efad1157d76-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:13.667243 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.667242 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:13.667418 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.667257 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:13.667418 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.667269 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:13.667418 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.667280 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:13.669341 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.669275 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-envoy" pod "5775b1d1-5ece-4ebd-9582-2efad1157d76" (UID: "5775b1d1-5ece-4ebd-9582-2efad1157d76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:13.669341 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.669287 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-podinfo" pod "5775b1d1-5ece-4ebd-9582-2efad1157d76" (UID: "5775b1d1-5ece-4ebd-9582-2efad1157d76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:50:13.669341 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.669320 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-token" pod "5775b1d1-5ece-4ebd-9582-2efad1157d76" (UID: "5775b1d1-5ece-4ebd-9582-2efad1157d76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:50:13.673125 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.673097 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5775b1d1-5ece-4ebd-9582-2efad1157d76-kube-api-access-d2c54" pod "5775b1d1-5ece-4ebd-9582-2efad1157d76" (UID: "5775b1d1-5ece-4ebd-9582-2efad1157d76"). InnerVolumeSpecName "kube-api-access-d2c54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:50:13.769221 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.769192 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:13.769341 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.769225 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2c54\" (UniqueName: \"kubernetes.io/projected/5775b1d1-5ece-4ebd-9582-2efad1157d76-kube-api-access-d2c54\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:13.769341 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.769243 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:13.769341 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.769256 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5775b1d1-5ece-4ebd-9582-2efad1157d76-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:13.926083 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.925965 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" event={"ID":"5775b1d1-5ece-4ebd-9582-2efad1157d76","Type":"ContainerDied","Data":"0141c4d2b322b80eb4944c62cbc39c9954ae6b426d3a04accbcef90fc6f20d44"} Oct 31 19:50:13.926766 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.926426 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5" Oct 31 19:50:13.928158 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.926698 4910 scope.go:122] "RemoveContainer" containerID="1fcd0786951b5e2577d527f85a080de437a29c6a8aafba18acda0e5c08dbd5ed" Oct 31 19:50:13.933100 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.932859 4910 generic.go:356] "Generic (PLEG): container finished" podID="4ca75657-04c9-4283-9059-220e69657f7c" containerID="4a3cea06a4855de38ba869100c1ccd037ca61a071422c740548fcc7204a8f628" exitCode=0 Oct 31 19:50:13.933100 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.933078 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" event={"ID":"4ca75657-04c9-4283-9059-220e69657f7c","Type":"ContainerDied","Data":"4a3cea06a4855de38ba869100c1ccd037ca61a071422c740548fcc7204a8f628"} Oct 31 19:50:13.947117 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.947078 4910 scope.go:122] "RemoveContainer" containerID="19c829415c183e4e9f2ad1ef1a25777173810df061bbd00f5d627fb439bf6d37" Oct 31 19:50:13.966042 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.966010 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5"] Oct 31 19:50:13.974854 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:13.974383 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-gj6s5"] Oct 31 19:50:14.057654 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.057616 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:50:14.177983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.177733 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ca75657-04c9-4283-9059-220e69657f7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ca75657-04c9-4283-9059-220e69657f7c-istio-podinfo\") pod \"4ca75657-04c9-4283-9059-220e69657f7c\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " Oct 31 19:50:14.179345 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.179319 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-credential-socket\") pod \"4ca75657-04c9-4283-9059-220e69657f7c\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " Oct 31 19:50:14.179969 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.179949 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ca75657-04c9-4283-9059-220e69657f7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ca75657-04c9-4283-9059-220e69657f7c-istiod-ca-cert\") pod \"4ca75657-04c9-4283-9059-220e69657f7c\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " Oct 31 19:50:14.180837 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.180816 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca75657-04c9-4283-9059-220e69657f7c-istio-token\" (UniqueName: \"kubernetes.io/projected/4ca75657-04c9-4283-9059-220e69657f7c-istio-token\") pod \"4ca75657-04c9-4283-9059-220e69657f7c\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " Oct 31 19:50:14.181589 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.181566 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-istio-data\") pod \"4ca75657-04c9-4283-9059-220e69657f7c\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " Oct 31 19:50:14.183656 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.183634 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-workload-certs\") pod \"4ca75657-04c9-4283-9059-220e69657f7c\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " Oct 31 19:50:14.186612 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.179784 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-credential-socket" pod "4ca75657-04c9-4283-9059-220e69657f7c" (UID: "4ca75657-04c9-4283-9059-220e69657f7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:14.186612 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.180756 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca75657-04c9-4283-9059-220e69657f7c-istiod-ca-cert" pod "4ca75657-04c9-4283-9059-220e69657f7c" (UID: "4ca75657-04c9-4283-9059-220e69657f7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:50:14.186612 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.181085 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ca75657-04c9-4283-9059-220e69657f7c-istio-podinfo" pod "4ca75657-04c9-4283-9059-220e69657f7c" (UID: "4ca75657-04c9-4283-9059-220e69657f7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:50:14.186612 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.182554 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-istio-data" pod "4ca75657-04c9-4283-9059-220e69657f7c" (UID: "4ca75657-04c9-4283-9059-220e69657f7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:14.186612 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.183876 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-workload-certs" pod "4ca75657-04c9-4283-9059-220e69657f7c" (UID: "4ca75657-04c9-4283-9059-220e69657f7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:14.186612 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.184026 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca75657-04c9-4283-9059-220e69657f7c-istio-token" pod "4ca75657-04c9-4283-9059-220e69657f7c" (UID: "4ca75657-04c9-4283-9059-220e69657f7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:50:14.187127 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.187108 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-workload-socket\") pod \"4ca75657-04c9-4283-9059-220e69657f7c\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " Oct 31 19:50:14.200998 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.196596 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-istio-envoy\") pod \"4ca75657-04c9-4283-9059-220e69657f7c\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " Oct 31 19:50:14.200998 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.196811 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca75657-04c9-4283-9059-220e69657f7c-kube-api-access-kg9lv\" (UniqueName: \"kubernetes.io/projected/4ca75657-04c9-4283-9059-220e69657f7c-kube-api-access-kg9lv\") pod \"4ca75657-04c9-4283-9059-220e69657f7c\" (UID: \"4ca75657-04c9-4283-9059-220e69657f7c\") " Oct 31 19:50:14.200998 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.198215 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ca75657-04c9-4283-9059-220e69657f7c-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:14.200998 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.198244 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:14.200998 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.198260 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ca75657-04c9-4283-9059-220e69657f7c-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:14.200998 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.198275 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ca75657-04c9-4283-9059-220e69657f7c-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:14.200998 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.198288 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:14.200998 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.198301 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:14.200998 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.200655 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-workload-socket" pod "4ca75657-04c9-4283-9059-220e69657f7c" (UID: "4ca75657-04c9-4283-9059-220e69657f7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:14.236148 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.234390 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca75657-04c9-4283-9059-220e69657f7c-kube-api-access-kg9lv" pod "4ca75657-04c9-4283-9059-220e69657f7c" (UID: "4ca75657-04c9-4283-9059-220e69657f7c"). InnerVolumeSpecName "kube-api-access-kg9lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:50:14.239933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.239072 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-istio-envoy" pod "4ca75657-04c9-4283-9059-220e69657f7c" (UID: "4ca75657-04c9-4283-9059-220e69657f7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:14.299965 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.299925 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:14.299965 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.299964 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ca75657-04c9-4283-9059-220e69657f7c-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:14.300178 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.299979 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg9lv\" (UniqueName: \"kubernetes.io/projected/4ca75657-04c9-4283-9059-220e69657f7c-kube-api-access-kg9lv\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:14.944790 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.944758 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" Oct 31 19:50:14.944790 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.944781 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65" event={"ID":"4ca75657-04c9-4283-9059-220e69657f7c","Type":"ContainerDied","Data":"349c4bf5c4df522fdac0bd963080b381b5fbe75b5d6a798f2dc85edeb185e314"} Oct 31 19:50:14.945372 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:14.944839 4910 scope.go:122] "RemoveContainer" containerID="4a3cea06a4855de38ba869100c1ccd037ca61a071422c740548fcc7204a8f628" Oct 31 19:50:15.034242 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:15.034196 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65"] Oct 31 19:50:15.040010 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:15.039981 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-03dfee-openshift-default-5ffbd99665-6zd65"] Oct 31 19:50:15.760009 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:15.759977 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ca75657-04c9-4283-9059-220e69657f7c" path="/var/lib/kubelet/pods/4ca75657-04c9-4283-9059-220e69657f7c/volumes" Oct 31 19:50:15.760664 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:15.760645 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5775b1d1-5ece-4ebd-9582-2efad1157d76" path="/var/lib/kubelet/pods/5775b1d1-5ece-4ebd-9582-2efad1157d76/volumes" Oct 31 19:50:19.106296 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:19.106256 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cefa93db-2v5jk"] Oct 31 19:50:19.107457 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:19.107437 4910 cpu_manager.go:414] "containerMap: removing container" podUID="5775b1d1-5ece-4ebd-9582-2efad1157d76" containerName="istio-proxy" Oct 31 19:50:19.107457 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:19.107456 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5775b1d1-5ece-4ebd-9582-2efad1157d76" containerName="istio-proxy" Oct 31 19:50:19.107646 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:19.107537 4910 cpu_manager.go:414] "containerMap: removing container" podUID="4ca75657-04c9-4283-9059-220e69657f7c" containerName="istio-proxy" Oct 31 19:50:19.107646 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:19.107543 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca75657-04c9-4283-9059-220e69657f7c" containerName="istio-proxy" Oct 31 19:50:19.107646 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:19.107565 4910 cpu_manager.go:414] "containerMap: removing container" podUID="4ca75657-04c9-4283-9059-220e69657f7c" containerName="istio-proxy" Oct 31 19:50:19.107646 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:19.107571 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca75657-04c9-4283-9059-220e69657f7c" containerName="istio-proxy" Oct 31 19:50:19.107866 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:19.107685 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca75657-04c9-4283-9059-220e69657f7c" containerName="istio-proxy" Oct 31 19:50:19.107866 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:19.107695 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca75657-04c9-4283-9059-220e69657f7c" containerName="istio-proxy" Oct 31 19:50:19.107866 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:19.107706 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="5775b1d1-5ece-4ebd-9582-2efad1157d76" containerName="istio-proxy" Oct 31 19:50:19.111896 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:19.111864 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cefa93db-2v5jk" Oct 31 19:50:19.124167 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:19.124138 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:50:19.124440 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:19.124423 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:50:19.128144 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:19.128117 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cefa93db-2v5jk"] Oct 31 19:50:19.207404 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:19.207370 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1363646-d9cf-4748-a70d-76daf00aba42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cefa93db-2v5jk\" (UID: \"c1363646-d9cf-4748-a70d-76daf00aba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cefa93db-2v5jk" Oct 31 19:50:19.207731 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:19.207699 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1363646-d9cf-4748-a70d-76daf00aba42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cefa93db-2v5jk\" (UID: \"c1363646-d9cf-4748-a70d-76daf00aba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cefa93db-2v5jk" Oct 31 19:50:19.207873 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:19.207861 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ggc2\" (UniqueName: \"kubernetes.io/projected/c1363646-d9cf-4748-a70d-76daf00aba42-kube-api-access-5ggc2\") pod \"maas-api-revoke-sub-cefa93db-2v5jk\" (UID: \"c1363646-d9cf-4748-a70d-76daf00aba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cefa93db-2v5jk" Oct 31 19:50:19.309655 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:19.309611 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1363646-d9cf-4748-a70d-76daf00aba42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cefa93db-2v5jk\" (UID: \"c1363646-d9cf-4748-a70d-76daf00aba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cefa93db-2v5jk" Oct 31 19:50:19.309814 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:19.309787 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ggc2\" (UniqueName: \"kubernetes.io/projected/c1363646-d9cf-4748-a70d-76daf00aba42-kube-api-access-5ggc2\") pod \"maas-api-revoke-sub-cefa93db-2v5jk\" (UID: \"c1363646-d9cf-4748-a70d-76daf00aba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cefa93db-2v5jk" Oct 31 19:50:19.309978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:19.309957 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1363646-d9cf-4748-a70d-76daf00aba42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cefa93db-2v5jk\" (UID: \"c1363646-d9cf-4748-a70d-76daf00aba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cefa93db-2v5jk" Oct 31 19:50:19.310681 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:19.310652 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1363646-d9cf-4748-a70d-76daf00aba42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cefa93db-2v5jk\" (UID: \"c1363646-d9cf-4748-a70d-76daf00aba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cefa93db-2v5jk" Oct 31 19:50:19.312746 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:19.312715 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1363646-d9cf-4748-a70d-76daf00aba42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cefa93db-2v5jk\" (UID: \"c1363646-d9cf-4748-a70d-76daf00aba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cefa93db-2v5jk" Oct 31 19:50:19.323725 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:19.323699 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ggc2\" (UniqueName: \"kubernetes.io/projected/c1363646-d9cf-4748-a70d-76daf00aba42-kube-api-access-5ggc2\") pod \"maas-api-revoke-sub-cefa93db-2v5jk\" (UID: \"c1363646-d9cf-4748-a70d-76daf00aba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cefa93db-2v5jk" Oct 31 19:50:19.437403 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:19.437322 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cefa93db-2v5jk" Oct 31 19:50:20.496022 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:50:20.458687 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1363646_d9cf_4748_a70d_76daf00aba42.slice/crio-8e30307dcb480c1ce699eff95085e8ceb715f588c741426047fb9d50fe4aa546 WatchSource:0}: Error finding container 8e30307dcb480c1ce699eff95085e8ceb715f588c741426047fb9d50fe4aa546: Status 404 returned error can't find the container with id 8e30307dcb480c1ce699eff95085e8ceb715f588c741426047fb9d50fe4aa546 Oct 31 19:50:20.517186 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:20.516232 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cefa93db-2v5jk"] Oct 31 19:50:20.517186 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:20.516295 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-278c04-96x9q"] Oct 31 19:50:20.523017 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:20.522993 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-278c04-96x9q" Oct 31 19:50:20.525105 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:20.525066 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-278c04-96x9q"] Oct 31 19:50:20.638424 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:20.638388 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0753d9eb-565f-4b49-8412-310bebbd48fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-278c04-96x9q\" (UID: \"0753d9eb-565f-4b49-8412-310bebbd48fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-278c04-96x9q" Oct 31 19:50:20.638619 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:20.638587 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0753d9eb-565f-4b49-8412-310bebbd48fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-278c04-96x9q\" (UID: \"0753d9eb-565f-4b49-8412-310bebbd48fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-278c04-96x9q" Oct 31 19:50:20.638713 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:20.638697 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw9kw\" (UniqueName: \"kubernetes.io/projected/0753d9eb-565f-4b49-8412-310bebbd48fc-kube-api-access-gw9kw\") pod \"maas-api-revoke-keys-e2e-inf-a-278c04-96x9q\" (UID: \"0753d9eb-565f-4b49-8412-310bebbd48fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-278c04-96x9q" Oct 31 19:50:20.740655 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:20.740562 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0753d9eb-565f-4b49-8412-310bebbd48fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-278c04-96x9q\" (UID: \"0753d9eb-565f-4b49-8412-310bebbd48fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-278c04-96x9q" Oct 31 19:50:20.740810 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:20.740784 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0753d9eb-565f-4b49-8412-310bebbd48fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-278c04-96x9q\" (UID: \"0753d9eb-565f-4b49-8412-310bebbd48fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-278c04-96x9q" Oct 31 19:50:20.740902 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:20.740886 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw9kw\" (UniqueName: \"kubernetes.io/projected/0753d9eb-565f-4b49-8412-310bebbd48fc-kube-api-access-gw9kw\") pod \"maas-api-revoke-keys-e2e-inf-a-278c04-96x9q\" (UID: \"0753d9eb-565f-4b49-8412-310bebbd48fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-278c04-96x9q" Oct 31 19:50:20.741414 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:20.741387 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0753d9eb-565f-4b49-8412-310bebbd48fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-278c04-96x9q\" (UID: \"0753d9eb-565f-4b49-8412-310bebbd48fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-278c04-96x9q" Oct 31 19:50:20.743423 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:20.743374 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0753d9eb-565f-4b49-8412-310bebbd48fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-278c04-96x9q\" (UID: \"0753d9eb-565f-4b49-8412-310bebbd48fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-278c04-96x9q" Oct 31 19:50:20.754213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:20.754191 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw9kw\" (UniqueName: \"kubernetes.io/projected/0753d9eb-565f-4b49-8412-310bebbd48fc-kube-api-access-gw9kw\") pod \"maas-api-revoke-keys-e2e-inf-a-278c04-96x9q\" (UID: \"0753d9eb-565f-4b49-8412-310bebbd48fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-278c04-96x9q" Oct 31 19:50:20.847082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:20.847050 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-278c04-96x9q" Oct 31 19:50:21.191285 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:21.189164 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-278c04-96x9q"] Oct 31 19:50:21.193487 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:50:21.193455 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0753d9eb_565f_4b49_8412_310bebbd48fc.slice/crio-a2e4dbbd1227f923da55e05ecd2433a688e80b0f908b1e7e7d673dc5dda267cc WatchSource:0}: Error finding container a2e4dbbd1227f923da55e05ecd2433a688e80b0f908b1e7e7d673dc5dda267cc: Status 404 returned error can't find the container with id a2e4dbbd1227f923da55e05ecd2433a688e80b0f908b1e7e7d673dc5dda267cc Oct 31 19:50:21.479328 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:21.479286 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-278c04-96x9q" event={"ID":"0753d9eb-565f-4b49-8412-310bebbd48fc","Type":"ContainerStarted","Data":"a2e4dbbd1227f923da55e05ecd2433a688e80b0f908b1e7e7d673dc5dda267cc"} Oct 31 19:50:21.481493 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:21.481458 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cefa93db-2v5jk" event={"ID":"c1363646-d9cf-4748-a70d-76daf00aba42","Type":"ContainerStarted","Data":"f5b4e67fbc6726b73cba2ac470afc154cb73562f8ae109a5623d0f9a04c9c8cf"} Oct 31 19:50:21.481646 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:21.481518 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cefa93db-2v5jk" event={"ID":"c1363646-d9cf-4748-a70d-76daf00aba42","Type":"ContainerStarted","Data":"8e30307dcb480c1ce699eff95085e8ceb715f588c741426047fb9d50fe4aa546"} Oct 31 19:50:21.661300 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:21.655062 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cefa93db-2v5jk" podStartSLOduration=2.655036171 podStartE2EDuration="2.655036171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:50:21.537260643 +0000 UTC m=+1855.900081015" watchObservedRunningTime="2026-10-31 19:50:21.655036171 +0000 UTC m=+1856.017856543" Oct 31 19:50:22.550003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:22.549887 4910 generic.go:356] "Generic (PLEG): container finished" podID="0753d9eb-565f-4b49-8412-310bebbd48fc" containerID="6feffa55d7f772b8968711118dd7252c8bb548c8f48218199a9f07945e64acce" exitCode=0 Oct 31 19:50:22.550247 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:22.550213 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-278c04-96x9q" event={"ID":"0753d9eb-565f-4b49-8412-310bebbd48fc","Type":"ContainerDied","Data":"6feffa55d7f772b8968711118dd7252c8bb548c8f48218199a9f07945e64acce"} Oct 31 19:50:22.572039 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:22.552654 4910 generic.go:356] "Generic (PLEG): container finished" podID="c1363646-d9cf-4748-a70d-76daf00aba42" containerID="f5b4e67fbc6726b73cba2ac470afc154cb73562f8ae109a5623d0f9a04c9c8cf" exitCode=0 Oct 31 19:50:22.572039 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:22.552839 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cefa93db-2v5jk" event={"ID":"c1363646-d9cf-4748-a70d-76daf00aba42","Type":"ContainerDied","Data":"f5b4e67fbc6726b73cba2ac470afc154cb73562f8ae109a5623d0f9a04c9c8cf"} Oct 31 19:50:23.872965 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:23.872896 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-278c04-96x9q" Oct 31 19:50:23.932618 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:23.926420 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cefa93db-2v5jk" Oct 31 19:50:23.968585 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:23.968544 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0753d9eb-565f-4b49-8412-310bebbd48fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0753d9eb-565f-4b49-8412-310bebbd48fc-maas-api-xks-ca\") pod \"0753d9eb-565f-4b49-8412-310bebbd48fc\" (UID: \"0753d9eb-565f-4b49-8412-310bebbd48fc\") " Oct 31 19:50:23.968861 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:23.968836 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0753d9eb-565f-4b49-8412-310bebbd48fc-kube-api-access-gw9kw\" (UniqueName: \"kubernetes.io/projected/0753d9eb-565f-4b49-8412-310bebbd48fc-kube-api-access-gw9kw\") pod \"0753d9eb-565f-4b49-8412-310bebbd48fc\" (UID: \"0753d9eb-565f-4b49-8412-310bebbd48fc\") " Oct 31 19:50:23.969006 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:23.968986 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0753d9eb-565f-4b49-8412-310bebbd48fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0753d9eb-565f-4b49-8412-310bebbd48fc-maas-api-openshift-ca\") pod \"0753d9eb-565f-4b49-8412-310bebbd48fc\" (UID: \"0753d9eb-565f-4b49-8412-310bebbd48fc\") " Oct 31 19:50:23.970150 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:23.970118 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0753d9eb-565f-4b49-8412-310bebbd48fc-maas-api-openshift-ca" pod "0753d9eb-565f-4b49-8412-310bebbd48fc" (UID: "0753d9eb-565f-4b49-8412-310bebbd48fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:50:23.976833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:23.974540 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0753d9eb-565f-4b49-8412-310bebbd48fc-maas-api-xks-ca" pod "0753d9eb-565f-4b49-8412-310bebbd48fc" (UID: "0753d9eb-565f-4b49-8412-310bebbd48fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:50:23.976833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:23.975076 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0753d9eb-565f-4b49-8412-310bebbd48fc-kube-api-access-gw9kw" pod "0753d9eb-565f-4b49-8412-310bebbd48fc" (UID: "0753d9eb-565f-4b49-8412-310bebbd48fc"). InnerVolumeSpecName "kube-api-access-gw9kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:50:24.070666 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:24.070633 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1363646-d9cf-4748-a70d-76daf00aba42-kube-api-access-5ggc2\" (UniqueName: \"kubernetes.io/projected/c1363646-d9cf-4748-a70d-76daf00aba42-kube-api-access-5ggc2\") pod \"c1363646-d9cf-4748-a70d-76daf00aba42\" (UID: \"c1363646-d9cf-4748-a70d-76daf00aba42\") " Oct 31 19:50:24.070863 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:24.070844 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1363646-d9cf-4748-a70d-76daf00aba42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1363646-d9cf-4748-a70d-76daf00aba42-maas-api-xks-ca\") pod \"c1363646-d9cf-4748-a70d-76daf00aba42\" (UID: \"c1363646-d9cf-4748-a70d-76daf00aba42\") " Oct 31 19:50:24.071040 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:24.071027 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1363646-d9cf-4748-a70d-76daf00aba42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1363646-d9cf-4748-a70d-76daf00aba42-maas-api-openshift-ca\") pod \"c1363646-d9cf-4748-a70d-76daf00aba42\" (UID: \"c1363646-d9cf-4748-a70d-76daf00aba42\") " Oct 31 19:50:24.071383 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:24.071356 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1363646-d9cf-4748-a70d-76daf00aba42-maas-api-openshift-ca" pod "c1363646-d9cf-4748-a70d-76daf00aba42" (UID: "c1363646-d9cf-4748-a70d-76daf00aba42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:50:24.071699 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:24.071679 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw9kw\" (UniqueName: \"kubernetes.io/projected/0753d9eb-565f-4b49-8412-310bebbd48fc-kube-api-access-gw9kw\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:24.071776 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:24.071701 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0753d9eb-565f-4b49-8412-310bebbd48fc-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:24.071776 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:24.071712 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0753d9eb-565f-4b49-8412-310bebbd48fc-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:24.071776 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:24.071721 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1363646-d9cf-4748-a70d-76daf00aba42-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:24.072951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:24.072923 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1363646-d9cf-4748-a70d-76daf00aba42-maas-api-xks-ca" pod "c1363646-d9cf-4748-a70d-76daf00aba42" (UID: "c1363646-d9cf-4748-a70d-76daf00aba42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:50:24.073084 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:24.073063 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1363646-d9cf-4748-a70d-76daf00aba42-kube-api-access-5ggc2" pod "c1363646-d9cf-4748-a70d-76daf00aba42" (UID: "c1363646-d9cf-4748-a70d-76daf00aba42"). InnerVolumeSpecName "kube-api-access-5ggc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:50:24.173780 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:24.173744 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1363646-d9cf-4748-a70d-76daf00aba42-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:24.173780 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:24.173776 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ggc2\" (UniqueName: \"kubernetes.io/projected/c1363646-d9cf-4748-a70d-76daf00aba42-kube-api-access-5ggc2\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:24.592971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:24.591397 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cefa93db-2v5jk" Oct 31 19:50:24.592971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:24.591430 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cefa93db-2v5jk" event={"ID":"c1363646-d9cf-4748-a70d-76daf00aba42","Type":"ContainerDied","Data":"8e30307dcb480c1ce699eff95085e8ceb715f588c741426047fb9d50fe4aa546"} Oct 31 19:50:24.592971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:24.591505 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e30307dcb480c1ce699eff95085e8ceb715f588c741426047fb9d50fe4aa546" Oct 31 19:50:24.595087 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:24.595054 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-278c04-96x9q" event={"ID":"0753d9eb-565f-4b49-8412-310bebbd48fc","Type":"ContainerDied","Data":"a2e4dbbd1227f923da55e05ecd2433a688e80b0f908b1e7e7d673dc5dda267cc"} Oct 31 19:50:24.595234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:24.595113 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2e4dbbd1227f923da55e05ecd2433a688e80b0f908b1e7e7d673dc5dda267cc" Oct 31 19:50:24.595234 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:24.595178 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-278c04-96x9q" Oct 31 19:50:30.417985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:30.408712 4910 scope.go:122] "RemoveContainer" containerID="2bd28068e2b8a12ee4ef46d2c10a26b4cfcc1ce391039696eb9433873539dafc" Oct 31 19:50:30.478020 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:30.471480 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj"] Oct 31 19:50:30.497392 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:30.497355 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c1363646-d9cf-4748-a70d-76daf00aba42" containerName="revoke-keys" Oct 31 19:50:30.497392 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:30.497396 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1363646-d9cf-4748-a70d-76daf00aba42" containerName="revoke-keys" Oct 31 19:50:30.497645 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:30.497445 4910 cpu_manager.go:414] "containerMap: removing container" podUID="0753d9eb-565f-4b49-8412-310bebbd48fc" containerName="revoke-keys" Oct 31 19:50:30.497645 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:30.497454 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0753d9eb-565f-4b49-8412-310bebbd48fc" containerName="revoke-keys" Oct 31 19:50:30.498003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:30.497938 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1363646-d9cf-4748-a70d-76daf00aba42" containerName="revoke-keys" Oct 31 19:50:30.498003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:30.497970 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="0753d9eb-565f-4b49-8412-310bebbd48fc" containerName="revoke-keys" Oct 31 19:50:30.507399 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:30.507370 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj" Oct 31 19:50:30.519005 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:30.518969 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj"] Oct 31 19:50:30.542633 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:30.542602 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:50:30.542824 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:30.542807 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:50:30.561927 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:30.559733 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50141a17-7d1b-478f-90eb-688e2132ca6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj\" (UID: \"50141a17-7d1b-478f-90eb-688e2132ca6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj" Oct 31 19:50:30.561927 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:30.559844 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50141a17-7d1b-478f-90eb-688e2132ca6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj\" (UID: \"50141a17-7d1b-478f-90eb-688e2132ca6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj" Oct 31 19:50:30.561927 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:30.559941 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx4vd\" (UniqueName: \"kubernetes.io/projected/50141a17-7d1b-478f-90eb-688e2132ca6e-kube-api-access-hx4vd\") pod \"maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj\" (UID: \"50141a17-7d1b-478f-90eb-688e2132ca6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj" Oct 31 19:50:30.662933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:30.662725 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50141a17-7d1b-478f-90eb-688e2132ca6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj\" (UID: \"50141a17-7d1b-478f-90eb-688e2132ca6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj" Oct 31 19:50:30.662933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:30.662834 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50141a17-7d1b-478f-90eb-688e2132ca6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj\" (UID: \"50141a17-7d1b-478f-90eb-688e2132ca6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj" Oct 31 19:50:30.662933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:30.662902 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx4vd\" (UniqueName: \"kubernetes.io/projected/50141a17-7d1b-478f-90eb-688e2132ca6e-kube-api-access-hx4vd\") pod \"maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj\" (UID: \"50141a17-7d1b-478f-90eb-688e2132ca6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj" Oct 31 19:50:30.667117 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:30.664143 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50141a17-7d1b-478f-90eb-688e2132ca6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj\" (UID: \"50141a17-7d1b-478f-90eb-688e2132ca6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj" Oct 31 19:50:30.670005 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:30.669573 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50141a17-7d1b-478f-90eb-688e2132ca6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj\" (UID: \"50141a17-7d1b-478f-90eb-688e2132ca6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj" Oct 31 19:50:30.678670 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:30.678642 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx4vd\" (UniqueName: \"kubernetes.io/projected/50141a17-7d1b-478f-90eb-688e2132ca6e-kube-api-access-hx4vd\") pod \"maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj\" (UID: \"50141a17-7d1b-478f-90eb-688e2132ca6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj" Oct 31 19:50:30.880084 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:30.880049 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj" Oct 31 19:50:31.614208 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:50:31.614173 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50141a17_7d1b_478f_90eb_688e2132ca6e.slice/crio-049977f965c09a7b2792f9c66cf93586ebb5979c3794dc6e4a6838cc167f1a18 WatchSource:0}: Error finding container 049977f965c09a7b2792f9c66cf93586ebb5979c3794dc6e4a6838cc167f1a18: Status 404 returned error can't find the container with id 049977f965c09a7b2792f9c66cf93586ebb5979c3794dc6e4a6838cc167f1a18 Oct 31 19:50:31.614625 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:31.614455 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj"] Oct 31 19:50:31.670713 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:31.670680 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj" event={"ID":"50141a17-7d1b-478f-90eb-688e2132ca6e","Type":"ContainerStarted","Data":"049977f965c09a7b2792f9c66cf93586ebb5979c3794dc6e4a6838cc167f1a18"} Oct 31 19:50:32.699933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:32.699423 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj" event={"ID":"50141a17-7d1b-478f-90eb-688e2132ca6e","Type":"ContainerStarted","Data":"4d844ca9272105dd46a24bf49e6250d25cfa4eee4cca9b5c49758fb8e5391215"} Oct 31 19:50:32.730307 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:32.730240 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj" podStartSLOduration=2.730218796 podStartE2EDuration="2.730218796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:50:32.719421139 +0000 UTC m=+1867.082241511" watchObservedRunningTime="2026-10-31 19:50:32.730218796 +0000 UTC m=+1867.093039170" Oct 31 19:50:33.709052 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:33.709017 4910 generic.go:356] "Generic (PLEG): container finished" podID="50141a17-7d1b-478f-90eb-688e2132ca6e" containerID="4d844ca9272105dd46a24bf49e6250d25cfa4eee4cca9b5c49758fb8e5391215" exitCode=0 Oct 31 19:50:33.709464 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:33.709106 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj" event={"ID":"50141a17-7d1b-478f-90eb-688e2132ca6e","Type":"ContainerDied","Data":"4d844ca9272105dd46a24bf49e6250d25cfa4eee4cca9b5c49758fb8e5391215"} Oct 31 19:50:34.915967 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:34.915895 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj" Oct 31 19:50:34.945394 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:34.945337 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h"] Oct 31 19:50:34.945670 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:34.945618 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h" podUID="329c14cb-5f70-4dc3-996e-71429c09d2ae" containerName="maas-api" containerID="cri-o://9a9a7334279751eabe032b67c8985843aae4135d7c632a3309cfc5a05862c839" gracePeriod=30 Oct 31 19:50:34.990370 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:34.989272 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50141a17-7d1b-478f-90eb-688e2132ca6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50141a17-7d1b-478f-90eb-688e2132ca6e-maas-api-openshift-ca\") pod \"50141a17-7d1b-478f-90eb-688e2132ca6e\" (UID: \"50141a17-7d1b-478f-90eb-688e2132ca6e\") " Oct 31 19:50:34.990370 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:34.989354 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50141a17-7d1b-478f-90eb-688e2132ca6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50141a17-7d1b-478f-90eb-688e2132ca6e-maas-api-xks-ca\") pod \"50141a17-7d1b-478f-90eb-688e2132ca6e\" (UID: \"50141a17-7d1b-478f-90eb-688e2132ca6e\") " Oct 31 19:50:34.990370 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:34.989410 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50141a17-7d1b-478f-90eb-688e2132ca6e-kube-api-access-hx4vd\" (UniqueName: \"kubernetes.io/projected/50141a17-7d1b-478f-90eb-688e2132ca6e-kube-api-access-hx4vd\") pod \"50141a17-7d1b-478f-90eb-688e2132ca6e\" (UID: \"50141a17-7d1b-478f-90eb-688e2132ca6e\") " Oct 31 19:50:34.991356 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:34.991325 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50141a17-7d1b-478f-90eb-688e2132ca6e-maas-api-openshift-ca" pod "50141a17-7d1b-478f-90eb-688e2132ca6e" (UID: "50141a17-7d1b-478f-90eb-688e2132ca6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:50:34.993548 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:34.993516 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50141a17-7d1b-478f-90eb-688e2132ca6e-kube-api-access-hx4vd" pod "50141a17-7d1b-478f-90eb-688e2132ca6e" (UID: "50141a17-7d1b-478f-90eb-688e2132ca6e"). InnerVolumeSpecName "kube-api-access-hx4vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:50:34.997133 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:34.994317 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50141a17-7d1b-478f-90eb-688e2132ca6e-maas-api-xks-ca" pod "50141a17-7d1b-478f-90eb-688e2132ca6e" (UID: "50141a17-7d1b-478f-90eb-688e2132ca6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:50:35.092364 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:35.092325 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50141a17-7d1b-478f-90eb-688e2132ca6e-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:35.092364 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:35.092364 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50141a17-7d1b-478f-90eb-688e2132ca6e-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:35.092578 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:35.092379 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx4vd\" (UniqueName: \"kubernetes.io/projected/50141a17-7d1b-478f-90eb-688e2132ca6e-kube-api-access-hx4vd\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:35.733988 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:35.713171 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-278c04-96x9q"] Oct 31 19:50:35.742977 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:35.741076 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj" event={"ID":"50141a17-7d1b-478f-90eb-688e2132ca6e","Type":"ContainerDied","Data":"049977f965c09a7b2792f9c66cf93586ebb5979c3794dc6e4a6838cc167f1a18"} Oct 31 19:50:35.742977 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:35.741150 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="049977f965c09a7b2792f9c66cf93586ebb5979c3794dc6e4a6838cc167f1a18" Oct 31 19:50:35.742977 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:35.741229 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj" Oct 31 19:50:35.777008 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:50:35.775454 4910 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-inf-a-278c04-96x9q\" is forbidden: User \"system:node:ip-10-0-142-116.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-142-116.us-east-2.compute.internal' and this object" podUID="0753d9eb-565f-4b49-8412-310bebbd48fc" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-278c04-96x9q" Oct 31 19:50:35.785673 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:35.785371 4910 generic.go:356] "Generic (PLEG): container finished" podID="329c14cb-5f70-4dc3-996e-71429c09d2ae" containerID="9a9a7334279751eabe032b67c8985843aae4135d7c632a3309cfc5a05862c839" exitCode=0 Oct 31 19:50:35.792877 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:35.792822 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h" event={"ID":"329c14cb-5f70-4dc3-996e-71429c09d2ae","Type":"ContainerDied","Data":"9a9a7334279751eabe032b67c8985843aae4135d7c632a3309cfc5a05862c839"} Oct 31 19:50:35.822307 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:35.822237 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-278c04-96x9q"] Oct 31 19:50:35.929348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:35.929319 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h" Oct 31 19:50:36.055023 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:36.054933 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/329c14cb-5f70-4dc3-996e-71429c09d2ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/329c14cb-5f70-4dc3-996e-71429c09d2ae-metrics-certs\") pod \"329c14cb-5f70-4dc3-996e-71429c09d2ae\" (UID: \"329c14cb-5f70-4dc3-996e-71429c09d2ae\") " Oct 31 19:50:36.055191 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:36.055029 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/329c14cb-5f70-4dc3-996e-71429c09d2ae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/329c14cb-5f70-4dc3-996e-71429c09d2ae-maas-api-tls\") pod \"329c14cb-5f70-4dc3-996e-71429c09d2ae\" (UID: \"329c14cb-5f70-4dc3-996e-71429c09d2ae\") " Oct 31 19:50:36.055191 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:36.055090 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/329c14cb-5f70-4dc3-996e-71429c09d2ae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/329c14cb-5f70-4dc3-996e-71429c09d2ae-openshift-service-ca\") pod \"329c14cb-5f70-4dc3-996e-71429c09d2ae\" (UID: \"329c14cb-5f70-4dc3-996e-71429c09d2ae\") " Oct 31 19:50:36.055391 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:36.055369 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/329c14cb-5f70-4dc3-996e-71429c09d2ae-kube-api-access-bvck8\" (UniqueName: \"kubernetes.io/projected/329c14cb-5f70-4dc3-996e-71429c09d2ae-kube-api-access-bvck8\") pod \"329c14cb-5f70-4dc3-996e-71429c09d2ae\" (UID: \"329c14cb-5f70-4dc3-996e-71429c09d2ae\") " Oct 31 19:50:36.055538 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:36.055512 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/329c14cb-5f70-4dc3-996e-71429c09d2ae-openshift-service-ca" pod "329c14cb-5f70-4dc3-996e-71429c09d2ae" (UID: "329c14cb-5f70-4dc3-996e-71429c09d2ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:50:36.056544 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:36.056520 4910 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/329c14cb-5f70-4dc3-996e-71429c09d2ae-openshift-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:36.057437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:36.057405 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/329c14cb-5f70-4dc3-996e-71429c09d2ae-maas-api-tls" pod "329c14cb-5f70-4dc3-996e-71429c09d2ae" (UID: "329c14cb-5f70-4dc3-996e-71429c09d2ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:50:36.057846 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:36.057784 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/329c14cb-5f70-4dc3-996e-71429c09d2ae-metrics-certs" pod "329c14cb-5f70-4dc3-996e-71429c09d2ae" (UID: "329c14cb-5f70-4dc3-996e-71429c09d2ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:50:36.058489 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:36.058461 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/329c14cb-5f70-4dc3-996e-71429c09d2ae-kube-api-access-bvck8" pod "329c14cb-5f70-4dc3-996e-71429c09d2ae" (UID: "329c14cb-5f70-4dc3-996e-71429c09d2ae"). InnerVolumeSpecName "kube-api-access-bvck8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:50:36.158455 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:36.158414 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/329c14cb-5f70-4dc3-996e-71429c09d2ae-maas-api-tls\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:36.158455 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:36.158455 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvck8\" (UniqueName: \"kubernetes.io/projected/329c14cb-5f70-4dc3-996e-71429c09d2ae-kube-api-access-bvck8\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:36.158700 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:36.158470 4910 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/329c14cb-5f70-4dc3-996e-71429c09d2ae-metrics-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:36.796178 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:36.796144 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h" Oct 31 19:50:36.796385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:36.796183 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h" event={"ID":"329c14cb-5f70-4dc3-996e-71429c09d2ae","Type":"ContainerDied","Data":"046e89a9c4737dff29ca62a6177bbcf944d48f3fe08fdb36472e09df9485a0fc"} Oct 31 19:50:36.796385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:36.796257 4910 scope.go:122] "RemoveContainer" containerID="9a9a7334279751eabe032b67c8985843aae4135d7c632a3309cfc5a05862c839" Oct 31 19:50:36.829385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:36.829355 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h"] Oct 31 19:50:36.836834 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:36.836807 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-278c04-5d8fb9dbdb-pn86h"] Oct 31 19:50:37.082972 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.082940 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf"] Oct 31 19:50:37.083711 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.083695 4910 cpu_manager.go:414] "containerMap: removing container" podUID="50141a17-7d1b-478f-90eb-688e2132ca6e" containerName="revoke-keys" Oct 31 19:50:37.083757 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.083712 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50141a17-7d1b-478f-90eb-688e2132ca6e" containerName="revoke-keys" Oct 31 19:50:37.083757 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.083748 4910 cpu_manager.go:414] "containerMap: removing container" podUID="329c14cb-5f70-4dc3-996e-71429c09d2ae" containerName="maas-api" Oct 31 19:50:37.083757 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.083753 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="329c14cb-5f70-4dc3-996e-71429c09d2ae" containerName="maas-api" Oct 31 19:50:37.083867 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.083852 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="329c14cb-5f70-4dc3-996e-71429c09d2ae" containerName="maas-api" Oct 31 19:50:37.083904 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.083867 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="50141a17-7d1b-478f-90eb-688e2132ca6e" containerName="revoke-keys" Oct 31 19:50:37.089097 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.089068 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.094368 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.094342 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx"] Oct 31 19:50:37.095052 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.095035 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-754e0a-openshift-default-dockercfg-t75h6\"" Oct 31 19:50:37.098739 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.098718 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.112738 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.112713 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf"] Oct 31 19:50:37.120244 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.120216 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx"] Oct 31 19:50:37.171059 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.171027 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea7fff1d-8kmst"] Oct 31 19:50:37.175789 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.175767 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7fff1d-8kmst" Oct 31 19:50:37.177315 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.177290 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-credential-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.177422 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.177348 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljp8q\" (UniqueName: \"kubernetes.io/projected/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-kube-api-access-ljp8q\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.177470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.177436 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-workload-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.177528 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.177494 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-podinfo\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.177625 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.177590 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-data\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.177747 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.177648 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-podinfo\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.177747 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.177720 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-token\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.177747 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.177742 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kx4b\" (UniqueName: \"kubernetes.io/projected/1f13c4cd-264c-44fc-814b-9dee39280ca1-kube-api-access-9kx4b\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.178011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.177762 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istiod-ca-cert\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.178011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.177873 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-workload-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.178011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.177902 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-data\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.178174 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.178012 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-credential-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.178174 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.178059 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-envoy\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.178174 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.178116 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f13c4cd-264c-44fc-814b-9dee39280ca1-istiod-ca-cert\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.178356 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.178211 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-token\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.178356 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.178303 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-workload-certs\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.178468 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.178369 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-envoy\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.178468 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.178412 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-workload-certs\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.185961 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.185901 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea7fff1d-8kmst"] Oct 31 19:50:37.197061 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.197035 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:50:37.197174 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.197066 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:50:37.280203 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.280168 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-credential-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.280410 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.280225 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljp8q\" (UniqueName: \"kubernetes.io/projected/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-kube-api-access-ljp8q\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.280410 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.280251 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-workload-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.280410 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.280283 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-podinfo\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.280410 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.280320 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-data\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.280410 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.280349 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-podinfo\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.280410 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.280399 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-token\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.280752 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.280432 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kx4b\" (UniqueName: \"kubernetes.io/projected/1f13c4cd-264c-44fc-814b-9dee39280ca1-kube-api-access-9kx4b\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.280752 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.280462 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istiod-ca-cert\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.280752 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.280517 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-workload-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.280752 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.280544 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-data\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.280752 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.280674 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-credential-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.281055 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.280790 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-data\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.281055 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.280890 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-data\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.281055 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.280958 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d44421ef-3970-47ca-8407-f2e75c7b9372-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea7fff1d-8kmst\" (UID: \"d44421ef-3970-47ca-8407-f2e75c7b9372\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7fff1d-8kmst" Oct 31 19:50:37.281055 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.281007 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-credential-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.281283 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.281076 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-envoy\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.281283 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.281090 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-workload-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.281283 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.281155 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f13c4cd-264c-44fc-814b-9dee39280ca1-istiod-ca-cert\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.281283 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.281221 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xvtw\" (UniqueName: \"kubernetes.io/projected/d44421ef-3970-47ca-8407-f2e75c7b9372-kube-api-access-4xvtw\") pod \"maas-api-revoke-sub-ea7fff1d-8kmst\" (UID: \"d44421ef-3970-47ca-8407-f2e75c7b9372\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7fff1d-8kmst" Oct 31 19:50:37.281283 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.281277 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-token\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.281555 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.281349 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-workload-certs\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.281555 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.281358 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istiod-ca-cert\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.281555 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.281357 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-credential-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.281716 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.281652 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-workload-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.281716 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.281656 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-workload-certs\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.281830 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.281806 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d44421ef-3970-47ca-8407-f2e75c7b9372-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea7fff1d-8kmst\" (UID: \"d44421ef-3970-47ca-8407-f2e75c7b9372\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7fff1d-8kmst" Oct 31 19:50:37.281887 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.281850 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-envoy\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.281957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.281896 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-workload-certs\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.282249 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.282219 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f13c4cd-264c-44fc-814b-9dee39280ca1-istiod-ca-cert\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.282357 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.282329 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-workload-certs\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.283540 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.283514 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-podinfo\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.283802 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.283778 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-envoy\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.284349 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.284324 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-envoy\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.284737 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.284717 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-podinfo\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.294351 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.294324 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kx4b\" (UniqueName: \"kubernetes.io/projected/1f13c4cd-264c-44fc-814b-9dee39280ca1-kube-api-access-9kx4b\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.294979 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.294903 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-token\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.295090 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.295051 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-token\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.295388 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.295369 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljp8q\" (UniqueName: \"kubernetes.io/projected/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-kube-api-access-ljp8q\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.383782 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.383687 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d44421ef-3970-47ca-8407-f2e75c7b9372-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea7fff1d-8kmst\" (UID: \"d44421ef-3970-47ca-8407-f2e75c7b9372\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7fff1d-8kmst" Oct 31 19:50:37.384035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.384011 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d44421ef-3970-47ca-8407-f2e75c7b9372-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea7fff1d-8kmst\" (UID: \"d44421ef-3970-47ca-8407-f2e75c7b9372\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7fff1d-8kmst" Oct 31 19:50:37.384166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.384144 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xvtw\" (UniqueName: \"kubernetes.io/projected/d44421ef-3970-47ca-8407-f2e75c7b9372-kube-api-access-4xvtw\") pod \"maas-api-revoke-sub-ea7fff1d-8kmst\" (UID: \"d44421ef-3970-47ca-8407-f2e75c7b9372\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7fff1d-8kmst" Oct 31 19:50:37.384562 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.384532 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d44421ef-3970-47ca-8407-f2e75c7b9372-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea7fff1d-8kmst\" (UID: \"d44421ef-3970-47ca-8407-f2e75c7b9372\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7fff1d-8kmst" Oct 31 19:50:37.387044 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.387024 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d44421ef-3970-47ca-8407-f2e75c7b9372-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea7fff1d-8kmst\" (UID: \"d44421ef-3970-47ca-8407-f2e75c7b9372\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7fff1d-8kmst" Oct 31 19:50:37.404135 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.404102 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:37.409484 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.409453 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xvtw\" (UniqueName: \"kubernetes.io/projected/d44421ef-3970-47ca-8407-f2e75c7b9372-kube-api-access-4xvtw\") pod \"maas-api-revoke-sub-ea7fff1d-8kmst\" (UID: \"d44421ef-3970-47ca-8407-f2e75c7b9372\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7fff1d-8kmst" Oct 31 19:50:37.418853 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.418823 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:37.502150 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:37.500691 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7fff1d-8kmst" Oct 31 19:50:38.067954 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:38.042853 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0753d9eb-565f-4b49-8412-310bebbd48fc" path="/var/lib/kubelet/pods/0753d9eb-565f-4b49-8412-310bebbd48fc/volumes" Oct 31 19:50:38.077737 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:38.069292 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="329c14cb-5f70-4dc3-996e-71429c09d2ae" path="/var/lib/kubelet/pods/329c14cb-5f70-4dc3-996e-71429c09d2ae/volumes" Oct 31 19:50:38.272103 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:38.271715 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx"] Oct 31 19:50:38.273375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:38.273176 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:50:38.273375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:38.273273 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:50:38.273375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:38.273322 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:50:38.466706 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:38.464950 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:50:38.466706 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:38.465047 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:50:38.466706 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:38.465086 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:50:38.487284 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:38.487171 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf"] Oct 31 19:50:38.497227 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:38.497189 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea7fff1d-8kmst"] Oct 31 19:50:39.372071 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:39.337016 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" event={"ID":"1f13c4cd-264c-44fc-814b-9dee39280ca1","Type":"ContainerStarted","Data":"8d85361589025d9bbf9c8670a78a7d8311566a9db402db386e1e07344ef11211"} Oct 31 19:50:39.372071 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:39.337082 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" event={"ID":"1f13c4cd-264c-44fc-814b-9dee39280ca1","Type":"ContainerStarted","Data":"5c130117c6bc3b95212d7d84cb01dfaf33954c74ba95753c790f2e8081ae5a2c"} Oct 31 19:50:39.376444 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:39.376411 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" event={"ID":"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08","Type":"ContainerStarted","Data":"2fbb5c4e654e2ab3c176bc4217fb13c5367740a612ab358e8658572099e62958"} Oct 31 19:50:39.376640 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:39.376627 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" event={"ID":"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08","Type":"ContainerStarted","Data":"b4fcf9d343870a8d8c4f10d75edce593112f3cf9da5f844ac1ba6358088495b0"} Oct 31 19:50:39.380804 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:39.380776 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7fff1d-8kmst" event={"ID":"d44421ef-3970-47ca-8407-f2e75c7b9372","Type":"ContainerStarted","Data":"a510a1ed1b0c4fffe466c3ad147988f645d85aa797ec1402348dbb1174bf9ad3"} Oct 31 19:50:39.381982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:39.381959 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7fff1d-8kmst" event={"ID":"d44421ef-3970-47ca-8407-f2e75c7b9372","Type":"ContainerStarted","Data":"d5c5d4fae4d9602b7411513f219c9966a7856c889353ae1bdc99d4ba83cf0b19"} Oct 31 19:50:39.407569 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:39.407536 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:39.409804 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:39.409778 4910 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 19:50:39.410046 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:39.410017 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" podUID="1f13c4cd-264c-44fc-814b-9dee39280ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 19:50:39.443331 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:39.443284 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:39.462506 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:39.462459 4910 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 31 19:50:39.463053 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:39.463017 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" podUID="ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 31 19:50:39.555284 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:39.551875 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" podStartSLOduration=2.55185431 podStartE2EDuration="2.55185431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:50:39.416433651 +0000 UTC m=+1873.779254025" watchObservedRunningTime="2026-10-31 19:50:39.55185431 +0000 UTC m=+1873.914674684" Oct 31 19:50:39.612971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:39.607968 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7fff1d-8kmst" podStartSLOduration=2.607946007 podStartE2EDuration="2.607946007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:50:39.580120178 +0000 UTC m=+1873.942940541" watchObservedRunningTime="2026-10-31 19:50:39.607946007 +0000 UTC m=+1873.970766381" Oct 31 19:50:39.700476 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:39.700349 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" podStartSLOduration=2.700328165 podStartE2EDuration="2.700328165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:50:39.689472112 +0000 UTC m=+1874.052292549" watchObservedRunningTime="2026-10-31 19:50:39.700328165 +0000 UTC m=+1874.063148539" Oct 31 19:50:40.410947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:40.409306 4910 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 19:50:40.410947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:40.409367 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" podUID="1f13c4cd-264c-44fc-814b-9dee39280ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 19:50:40.421831 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:40.421792 4910 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 31 19:50:40.422041 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:40.421853 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" podUID="ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 31 19:50:40.438969 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:40.432728 4910 generic.go:356] "Generic (PLEG): container finished" podID="d44421ef-3970-47ca-8407-f2e75c7b9372" containerID="a510a1ed1b0c4fffe466c3ad147988f645d85aa797ec1402348dbb1174bf9ad3" exitCode=0 Oct 31 19:50:40.438969 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:40.434217 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7fff1d-8kmst" event={"ID":"d44421ef-3970-47ca-8407-f2e75c7b9372","Type":"ContainerDied","Data":"a510a1ed1b0c4fffe466c3ad147988f645d85aa797ec1402348dbb1174bf9ad3"} Oct 31 19:50:40.902938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:40.887375 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j"] Oct 31 19:50:41.308997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:41.307220 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:50:41.375146 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:41.319230 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:50:41.453932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:41.421835 4910 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 31 19:50:41.585993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:41.454905 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" podUID="ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 31 19:50:41.585993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:41.567820 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:50:41.664775 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:41.653113 4910 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 19:50:41.664775 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:41.653176 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" podUID="1f13c4cd-264c-44fc-814b-9dee39280ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 19:50:41.867042 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:41.847983 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:50:42.073428 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:42.073113 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:50:42.343932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:42.342607 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:50:42.426202 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:42.426162 4910 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 19:50:42.426514 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:42.426469 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" podUID="1f13c4cd-264c-44fc-814b-9dee39280ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 19:50:42.432133 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:42.432088 4910 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 31 19:50:42.432382 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:42.432345 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" podUID="ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 31 19:50:42.604147 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:42.604041 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:50:42.605571 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:42.605544 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:50:42.701167 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:42.698640 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 19:50:42.706966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:42.706865 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:50:42.844931 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:50:42.811301 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:50:42.844931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:42.816056 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7fff1d-8kmst" Oct 31 19:50:42.954011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:42.939696 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d44421ef-3970-47ca-8407-f2e75c7b9372-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d44421ef-3970-47ca-8407-f2e75c7b9372-maas-api-xks-ca\") pod \"d44421ef-3970-47ca-8407-f2e75c7b9372\" (UID: \"d44421ef-3970-47ca-8407-f2e75c7b9372\") " Oct 31 19:50:42.973939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:42.963939 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d44421ef-3970-47ca-8407-f2e75c7b9372-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d44421ef-3970-47ca-8407-f2e75c7b9372-maas-api-openshift-ca\") pod \"d44421ef-3970-47ca-8407-f2e75c7b9372\" (UID: \"d44421ef-3970-47ca-8407-f2e75c7b9372\") " Oct 31 19:50:42.979934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:42.978726 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d44421ef-3970-47ca-8407-f2e75c7b9372-kube-api-access-4xvtw\" (UniqueName: \"kubernetes.io/projected/d44421ef-3970-47ca-8407-f2e75c7b9372-kube-api-access-4xvtw\") pod \"d44421ef-3970-47ca-8407-f2e75c7b9372\" (UID: \"d44421ef-3970-47ca-8407-f2e75c7b9372\") " Oct 31 19:50:42.981289 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:42.981158 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44421ef-3970-47ca-8407-f2e75c7b9372-maas-api-openshift-ca" pod "d44421ef-3970-47ca-8407-f2e75c7b9372" (UID: "d44421ef-3970-47ca-8407-f2e75c7b9372"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:50:43.007013 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:43.006982 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7fff1d-8kmst" Oct 31 19:50:43.007258 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:43.007223 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7fff1d-8kmst" event={"ID":"d44421ef-3970-47ca-8407-f2e75c7b9372","Type":"ContainerDied","Data":"d5c5d4fae4d9602b7411513f219c9966a7856c889353ae1bdc99d4ba83cf0b19"} Oct 31 19:50:43.007322 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:43.007286 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5c5d4fae4d9602b7411513f219c9966a7856c889353ae1bdc99d4ba83cf0b19" Oct 31 19:50:43.010290 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:43.010139 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d44421ef-3970-47ca-8407-f2e75c7b9372-maas-api-xks-ca" pod "d44421ef-3970-47ca-8407-f2e75c7b9372" (UID: "d44421ef-3970-47ca-8407-f2e75c7b9372"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:50:43.012730 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:43.012693 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44421ef-3970-47ca-8407-f2e75c7b9372-kube-api-access-4xvtw" pod "d44421ef-3970-47ca-8407-f2e75c7b9372" (UID: "d44421ef-3970-47ca-8407-f2e75c7b9372"). InnerVolumeSpecName "kube-api-access-4xvtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:50:43.085521 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:43.085322 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d44421ef-3970-47ca-8407-f2e75c7b9372-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:43.085521 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:43.085522 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d44421ef-3970-47ca-8407-f2e75c7b9372-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:43.085795 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:43.085540 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xvtw\" (UniqueName: \"kubernetes.io/projected/d44421ef-3970-47ca-8407-f2e75c7b9372-kube-api-access-4xvtw\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:43.436941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:43.433466 4910 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 31 19:50:43.436941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:43.433539 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" podUID="ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 31 19:50:43.457933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:43.449473 4910 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 19:50:43.457933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:43.449733 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" podUID="1f13c4cd-264c-44fc-814b-9dee39280ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 19:50:44.427933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:44.407305 4910 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 19:50:44.427933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:44.407367 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" podUID="1f13c4cd-264c-44fc-814b-9dee39280ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 19:50:44.441935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:44.434753 4910 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 31 19:50:44.441935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:44.434809 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" podUID="ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 31 19:50:45.405815 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:45.405778 4910 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 19:50:45.406032 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:45.405846 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" podUID="1f13c4cd-264c-44fc-814b-9dee39280ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 19:50:45.436719 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:45.429421 4910 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 31 19:50:45.436719 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:45.429477 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" podUID="ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 31 19:50:45.896935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:45.892107 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j" podUID="09ee3f2a-332e-4c98-9b21-c76f3903b69f" containerName="payload-pre-processing" containerID="cri-o://cd2371917831ea70df2741d58567ef8a93339d01d1b41ae865a4415ed9c429d3" gracePeriod=25 Oct 31 19:50:46.153446 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.153334 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj"] Oct 31 19:50:46.154696 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.154056 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" podUID="499b41f2-15df-4636-abbc-35fa6f8a164a" containerName="istio-proxy" containerID="cri-o://853d31bcc99e84aa887bf15f15988fde746f71055e45e99b9091cf9f3f22322f" gracePeriod=30 Oct 31 19:50:46.169204 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.168788 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch"] Oct 31 19:50:46.169204 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.169016 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" podUID="199b78b1-0dcc-44df-a25b-9c401a86ad70" containerName="istio-proxy" containerID="cri-o://64490afe30f522a499652851d571209250fcba5b54658150b7da616bfad14564" gracePeriod=30 Oct 31 19:50:46.176893 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.176733 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql"] Oct 31 19:50:46.177888 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.177864 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d44421ef-3970-47ca-8407-f2e75c7b9372" containerName="revoke-keys" Oct 31 19:50:46.178069 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.177891 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d44421ef-3970-47ca-8407-f2e75c7b9372" containerName="revoke-keys" Oct 31 19:50:46.178176 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.178159 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d44421ef-3970-47ca-8407-f2e75c7b9372" containerName="revoke-keys" Oct 31 19:50:46.182970 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.182947 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql" Oct 31 19:50:46.217134 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.217103 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql"] Oct 31 19:50:46.266210 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.266174 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:50:46.266558 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.266524 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:50:46.271157 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.271127 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b97ba48f-5c91-4007-bb09-a54f35264beb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql\" (UID: \"b97ba48f-5c91-4007-bb09-a54f35264beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql" Oct 31 19:50:46.271298 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.271191 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2kgk\" (UniqueName: \"kubernetes.io/projected/b97ba48f-5c91-4007-bb09-a54f35264beb-kube-api-access-s2kgk\") pod \"maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql\" (UID: \"b97ba48f-5c91-4007-bb09-a54f35264beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql" Oct 31 19:50:46.271422 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.271399 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b97ba48f-5c91-4007-bb09-a54f35264beb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql\" (UID: \"b97ba48f-5c91-4007-bb09-a54f35264beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql" Oct 31 19:50:46.385929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.374251 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b97ba48f-5c91-4007-bb09-a54f35264beb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql\" (UID: \"b97ba48f-5c91-4007-bb09-a54f35264beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql" Oct 31 19:50:46.385929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.374550 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b97ba48f-5c91-4007-bb09-a54f35264beb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql\" (UID: \"b97ba48f-5c91-4007-bb09-a54f35264beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql" Oct 31 19:50:46.385929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.374602 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2kgk\" (UniqueName: \"kubernetes.io/projected/b97ba48f-5c91-4007-bb09-a54f35264beb-kube-api-access-s2kgk\") pod \"maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql\" (UID: \"b97ba48f-5c91-4007-bb09-a54f35264beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql" Oct 31 19:50:46.385929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.375785 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b97ba48f-5c91-4007-bb09-a54f35264beb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql\" (UID: \"b97ba48f-5c91-4007-bb09-a54f35264beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql" Oct 31 19:50:46.426119 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.402445 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b97ba48f-5c91-4007-bb09-a54f35264beb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql\" (UID: \"b97ba48f-5c91-4007-bb09-a54f35264beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql" Oct 31 19:50:46.427614 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.427197 4910 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 31 19:50:46.427614 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.427266 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" podUID="ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 31 19:50:46.429118 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.428962 4910 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Oct 31 19:50:46.429352 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.429326 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" podUID="1f13c4cd-264c-44fc-814b-9dee39280ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Oct 31 19:50:46.437270 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.437202 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2kgk\" (UniqueName: \"kubernetes.io/projected/b97ba48f-5c91-4007-bb09-a54f35264beb-kube-api-access-s2kgk\") pod \"maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql\" (UID: \"b97ba48f-5c91-4007-bb09-a54f35264beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql" Oct 31 19:50:46.503753 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.503716 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql" Oct 31 19:50:46.868745 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:46.868714 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql"] Oct 31 19:50:46.870803 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:50:46.870778 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97ba48f_5c91_4007_bb09_a54f35264beb.slice/crio-410e699280f5d96686b32872cc3b7ec3260af6fc20c5414d7d31d3fc28c5146f WatchSource:0}: Error finding container 410e699280f5d96686b32872cc3b7ec3260af6fc20c5414d7d31d3fc28c5146f: Status 404 returned error can't find the container with id 410e699280f5d96686b32872cc3b7ec3260af6fc20c5414d7d31d3fc28c5146f Oct 31 19:50:47.358616 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:47.358582 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql" event={"ID":"b97ba48f-5c91-4007-bb09-a54f35264beb","Type":"ContainerStarted","Data":"410e699280f5d96686b32872cc3b7ec3260af6fc20c5414d7d31d3fc28c5146f"} Oct 31 19:50:47.413597 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:47.413533 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:47.414671 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:47.414645 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:47.416577 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:47.416556 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:50:47.425068 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:47.425045 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:47.425354 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:47.425336 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:47.426192 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:47.426170 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:50:48.369638 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:48.369595 4910 generic.go:356] "Generic (PLEG): container finished" podID="b97ba48f-5c91-4007-bb09-a54f35264beb" containerID="f5eb1675d3d5a826d29ccdccf56cf13206179c9e005867d6f166c5bf55ab7362" exitCode=0 Oct 31 19:50:48.370059 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:48.369666 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql" event={"ID":"b97ba48f-5c91-4007-bb09-a54f35264beb","Type":"ContainerDied","Data":"f5eb1675d3d5a826d29ccdccf56cf13206179c9e005867d6f166c5bf55ab7362"} Oct 31 19:50:49.164518 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:49.164484 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b85fa0a2-68tnp"] Oct 31 19:50:49.169331 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:49.169308 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85fa0a2-68tnp" Oct 31 19:50:49.192585 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:49.192556 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b85fa0a2-68tnp"] Oct 31 19:50:49.213526 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:49.213486 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lwwm\" (UniqueName: \"kubernetes.io/projected/76ca271f-5d14-4807-928d-a4c5c23debc4-kube-api-access-9lwwm\") pod \"maas-api-revoke-sub-b85fa0a2-68tnp\" (UID: \"76ca271f-5d14-4807-928d-a4c5c23debc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85fa0a2-68tnp" Oct 31 19:50:49.213777 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:49.213755 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ca271f-5d14-4807-928d-a4c5c23debc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b85fa0a2-68tnp\" (UID: \"76ca271f-5d14-4807-928d-a4c5c23debc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85fa0a2-68tnp" Oct 31 19:50:49.214071 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:49.214044 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ca271f-5d14-4807-928d-a4c5c23debc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b85fa0a2-68tnp\" (UID: \"76ca271f-5d14-4807-928d-a4c5c23debc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85fa0a2-68tnp" Oct 31 19:50:49.307549 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:49.307515 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd"] Oct 31 19:50:49.307761 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:49.307727 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" podUID="706169ef-5dc5-4533-86af-fd02de456ab0" containerName="istio-proxy" containerID="cri-o://cc37f5426fae5d07363bc461faf8a4d307bc71f4bdb50f30f785a298622423b0" gracePeriod=30 Oct 31 19:50:49.315550 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:49.315518 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ca271f-5d14-4807-928d-a4c5c23debc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b85fa0a2-68tnp\" (UID: \"76ca271f-5d14-4807-928d-a4c5c23debc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85fa0a2-68tnp" Oct 31 19:50:49.316971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:49.315738 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ca271f-5d14-4807-928d-a4c5c23debc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b85fa0a2-68tnp\" (UID: \"76ca271f-5d14-4807-928d-a4c5c23debc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85fa0a2-68tnp" Oct 31 19:50:49.316971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:49.315823 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lwwm\" (UniqueName: \"kubernetes.io/projected/76ca271f-5d14-4807-928d-a4c5c23debc4-kube-api-access-9lwwm\") pod \"maas-api-revoke-sub-b85fa0a2-68tnp\" (UID: \"76ca271f-5d14-4807-928d-a4c5c23debc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85fa0a2-68tnp" Oct 31 19:50:49.316971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:49.316519 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ca271f-5d14-4807-928d-a4c5c23debc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b85fa0a2-68tnp\" (UID: \"76ca271f-5d14-4807-928d-a4c5c23debc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85fa0a2-68tnp" Oct 31 19:50:49.318223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:49.318180 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ca271f-5d14-4807-928d-a4c5c23debc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b85fa0a2-68tnp\" (UID: \"76ca271f-5d14-4807-928d-a4c5c23debc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85fa0a2-68tnp" Oct 31 19:50:49.345068 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:49.345031 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lwwm\" (UniqueName: \"kubernetes.io/projected/76ca271f-5d14-4807-928d-a4c5c23debc4-kube-api-access-9lwwm\") pod \"maas-api-revoke-sub-b85fa0a2-68tnp\" (UID: \"76ca271f-5d14-4807-928d-a4c5c23debc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85fa0a2-68tnp" Oct 31 19:50:49.484280 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:49.484173 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85fa0a2-68tnp" Oct 31 19:50:49.871505 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:49.869936 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4"] Oct 31 19:50:49.871505 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:49.870182 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4" podUID="01593536-a296-4333-a1a2-cca5cedeb8f9" containerName="maas-api" containerID="cri-o://b8adbe1c5e2ab4f141f43373af7c0affc7682f476e812bff205d349731ad9ca4" gracePeriod=30 Oct 31 19:50:49.958783 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:49.958745 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4" podUID="01593536-a296-4333-a1a2-cca5cedeb8f9" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.175:8443/health\": dial tcp 10.132.0.175:8443: connect: connection refused" Oct 31 19:50:50.000867 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:50.000483 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql" Oct 31 19:50:50.061528 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:50.049269 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b97ba48f-5c91-4007-bb09-a54f35264beb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b97ba48f-5c91-4007-bb09-a54f35264beb-maas-api-openshift-ca\") pod \"b97ba48f-5c91-4007-bb09-a54f35264beb\" (UID: \"b97ba48f-5c91-4007-bb09-a54f35264beb\") " Oct 31 19:50:50.061528 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:50.049599 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b97ba48f-5c91-4007-bb09-a54f35264beb-kube-api-access-s2kgk\" (UniqueName: \"kubernetes.io/projected/b97ba48f-5c91-4007-bb09-a54f35264beb-kube-api-access-s2kgk\") pod \"b97ba48f-5c91-4007-bb09-a54f35264beb\" (UID: \"b97ba48f-5c91-4007-bb09-a54f35264beb\") " Oct 31 19:50:50.061528 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:50.049639 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b97ba48f-5c91-4007-bb09-a54f35264beb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b97ba48f-5c91-4007-bb09-a54f35264beb-maas-api-xks-ca\") pod \"b97ba48f-5c91-4007-bb09-a54f35264beb\" (UID: \"b97ba48f-5c91-4007-bb09-a54f35264beb\") " Oct 31 19:50:50.061528 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:50.053295 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b97ba48f-5c91-4007-bb09-a54f35264beb-maas-api-xks-ca" pod "b97ba48f-5c91-4007-bb09-a54f35264beb" (UID: "b97ba48f-5c91-4007-bb09-a54f35264beb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:50:50.061528 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:50.053635 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b97ba48f-5c91-4007-bb09-a54f35264beb-maas-api-openshift-ca" pod "b97ba48f-5c91-4007-bb09-a54f35264beb" (UID: "b97ba48f-5c91-4007-bb09-a54f35264beb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:50:50.186968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:50.173589 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b97ba48f-5c91-4007-bb09-a54f35264beb-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:50.186968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:50.173628 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b97ba48f-5c91-4007-bb09-a54f35264beb-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:50.294729 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:50.294540 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97ba48f-5c91-4007-bb09-a54f35264beb-kube-api-access-s2kgk" pod "b97ba48f-5c91-4007-bb09-a54f35264beb" (UID: "b97ba48f-5c91-4007-bb09-a54f35264beb"). InnerVolumeSpecName "kube-api-access-s2kgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:50:50.486588 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:50.486414 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2kgk\" (UniqueName: \"kubernetes.io/projected/b97ba48f-5c91-4007-bb09-a54f35264beb-kube-api-access-s2kgk\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:50.560054 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:50.559991 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj"] Oct 31 19:50:50.570618 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:50.570527 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4e312-7j2kj"] Oct 31 19:50:50.646121 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:50.646080 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b85fa0a2-68tnp"] Oct 31 19:50:50.657470 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:50:50.652593 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ca271f_5d14_4807_928d_a4c5c23debc4.slice/crio-ff47ef101da9e840eb092b6803640bf46f3c2d8fd2a03fa2d241cbb7ff88dad2 WatchSource:0}: Error finding container ff47ef101da9e840eb092b6803640bf46f3c2d8fd2a03fa2d241cbb7ff88dad2: Status 404 returned error can't find the container with id ff47ef101da9e840eb092b6803640bf46f3c2d8fd2a03fa2d241cbb7ff88dad2 Oct 31 19:50:50.689854 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:50.688237 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql" event={"ID":"b97ba48f-5c91-4007-bb09-a54f35264beb","Type":"ContainerDied","Data":"410e699280f5d96686b32872cc3b7ec3260af6fc20c5414d7d31d3fc28c5146f"} Oct 31 19:50:50.689854 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:50.688330 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="410e699280f5d96686b32872cc3b7ec3260af6fc20c5414d7d31d3fc28c5146f" Oct 31 19:50:50.689854 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:50.688442 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql" Oct 31 19:50:51.238100 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.235052 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w"] Oct 31 19:50:51.238100 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.235360 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" podUID="be3bf6bf-4863-4489-b924-3de83b589c8e" containerName="istio-proxy" containerID="cri-o://eb4d5dc8612c55538b4fe5a87db69d023da6d807ad5f4617dbbf7d0858497bb3" gracePeriod=30 Oct 31 19:50:51.250989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.250943 4910 generic.go:356] "Generic (PLEG): container finished" podID="01593536-a296-4333-a1a2-cca5cedeb8f9" containerID="b8adbe1c5e2ab4f141f43373af7c0affc7682f476e812bff205d349731ad9ca4" exitCode=0 Oct 31 19:50:51.251280 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.251259 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4" event={"ID":"01593536-a296-4333-a1a2-cca5cedeb8f9","Type":"ContainerDied","Data":"b8adbe1c5e2ab4f141f43373af7c0affc7682f476e812bff205d349731ad9ca4"} Oct 31 19:50:51.405646 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:50:51.405604 4910 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97ba48f_5c91_4007_bb09_a54f35264beb.slice\": RecentStats: unable to find data in memory cache]" Oct 31 19:50:51.443461 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.443434 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4" Oct 31 19:50:51.471995 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.471885 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01593536-a296-4333-a1a2-cca5cedeb8f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/01593536-a296-4333-a1a2-cca5cedeb8f9-metrics-certs\") pod \"01593536-a296-4333-a1a2-cca5cedeb8f9\" (UID: \"01593536-a296-4333-a1a2-cca5cedeb8f9\") " Oct 31 19:50:51.472149 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.472010 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01593536-a296-4333-a1a2-cca5cedeb8f9-kube-api-access-hdqlw\" (UniqueName: \"kubernetes.io/projected/01593536-a296-4333-a1a2-cca5cedeb8f9-kube-api-access-hdqlw\") pod \"01593536-a296-4333-a1a2-cca5cedeb8f9\" (UID: \"01593536-a296-4333-a1a2-cca5cedeb8f9\") " Oct 31 19:50:51.472254 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.472233 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01593536-a296-4333-a1a2-cca5cedeb8f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01593536-a296-4333-a1a2-cca5cedeb8f9-maas-api-tls\") pod \"01593536-a296-4333-a1a2-cca5cedeb8f9\" (UID: \"01593536-a296-4333-a1a2-cca5cedeb8f9\") " Oct 31 19:50:51.472349 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.472315 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01593536-a296-4333-a1a2-cca5cedeb8f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01593536-a296-4333-a1a2-cca5cedeb8f9-openshift-service-ca\") pod \"01593536-a296-4333-a1a2-cca5cedeb8f9\" (UID: \"01593536-a296-4333-a1a2-cca5cedeb8f9\") " Oct 31 19:50:51.474019 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.473986 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01593536-a296-4333-a1a2-cca5cedeb8f9-openshift-service-ca" pod "01593536-a296-4333-a1a2-cca5cedeb8f9" (UID: "01593536-a296-4333-a1a2-cca5cedeb8f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:50:51.474638 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.474608 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01593536-a296-4333-a1a2-cca5cedeb8f9-kube-api-access-hdqlw" pod "01593536-a296-4333-a1a2-cca5cedeb8f9" (UID: "01593536-a296-4333-a1a2-cca5cedeb8f9"). InnerVolumeSpecName "kube-api-access-hdqlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:50:51.475948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.475906 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01593536-a296-4333-a1a2-cca5cedeb8f9-maas-api-tls" pod "01593536-a296-4333-a1a2-cca5cedeb8f9" (UID: "01593536-a296-4333-a1a2-cca5cedeb8f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:50:51.501614 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.501582 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01593536-a296-4333-a1a2-cca5cedeb8f9-metrics-certs" pod "01593536-a296-4333-a1a2-cca5cedeb8f9" (UID: "01593536-a296-4333-a1a2-cca5cedeb8f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:50:51.575280 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.575143 4910 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01593536-a296-4333-a1a2-cca5cedeb8f9-metrics-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:51.575513 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.575495 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdqlw\" (UniqueName: \"kubernetes.io/projected/01593536-a296-4333-a1a2-cca5cedeb8f9-kube-api-access-hdqlw\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:51.575613 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.575601 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01593536-a296-4333-a1a2-cca5cedeb8f9-maas-api-tls\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:51.575705 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.575693 4910 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01593536-a296-4333-a1a2-cca5cedeb8f9-openshift-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:51.680842 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.680811 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:50:51.684077 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.684053 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:50:51.766708 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.766585 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50141a17-7d1b-478f-90eb-688e2132ca6e" path="/var/lib/kubelet/pods/50141a17-7d1b-478f-90eb-688e2132ca6e/volumes" Oct 31 19:50:51.780310 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.780283 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-token\" (UniqueName: \"kubernetes.io/projected/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-token\") pod \"199b78b1-0dcc-44df-a25b-9c401a86ad70\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " Oct 31 19:50:51.781093 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.781069 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-data\") pod \"199b78b1-0dcc-44df-a25b-9c401a86ad70\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " Oct 31 19:50:51.781299 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.781281 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-envoy\") pod \"499b41f2-15df-4636-abbc-35fa6f8a164a\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " Oct 31 19:50:51.781463 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.781430 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-data\") pod \"499b41f2-15df-4636-abbc-35fa6f8a164a\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " Oct 31 19:50:51.781604 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.781587 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/199b78b1-0dcc-44df-a25b-9c401a86ad70-kube-api-access-6rbb6\" (UniqueName: \"kubernetes.io/projected/199b78b1-0dcc-44df-a25b-9c401a86ad70-kube-api-access-6rbb6\") pod \"199b78b1-0dcc-44df-a25b-9c401a86ad70\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " Oct 31 19:50:51.781802 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.781783 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-workload-socket\") pod \"499b41f2-15df-4636-abbc-35fa6f8a164a\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " Oct 31 19:50:51.782008 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.781989 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-credential-socket\") pod \"499b41f2-15df-4636-abbc-35fa6f8a164a\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " Oct 31 19:50:51.782156 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.782140 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-credential-socket\") pod \"199b78b1-0dcc-44df-a25b-9c401a86ad70\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " Oct 31 19:50:51.782366 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.782349 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-token\" (UniqueName: \"kubernetes.io/projected/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-token\") pod \"499b41f2-15df-4636-abbc-35fa6f8a164a\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " Oct 31 19:50:51.782493 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.782478 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-workload-certs\") pod \"499b41f2-15df-4636-abbc-35fa6f8a164a\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.782614 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-data" pod "499b41f2-15df-4636-abbc-35fa6f8a164a" (UID: "499b41f2-15df-4636-abbc-35fa6f8a164a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.782655 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/199b78b1-0dcc-44df-a25b-9c401a86ad70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/199b78b1-0dcc-44df-a25b-9c401a86ad70-istiod-ca-cert\") pod \"199b78b1-0dcc-44df-a25b-9c401a86ad70\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.782694 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/499b41f2-15df-4636-abbc-35fa6f8a164a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/499b41f2-15df-4636-abbc-35fa6f8a164a-istiod-ca-cert\") pod \"499b41f2-15df-4636-abbc-35fa6f8a164a\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.782735 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-envoy\") pod \"199b78b1-0dcc-44df-a25b-9c401a86ad70\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.782765 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-podinfo\") pod \"199b78b1-0dcc-44df-a25b-9c401a86ad70\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.782825 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-workload-socket" pod "499b41f2-15df-4636-abbc-35fa6f8a164a" (UID: "499b41f2-15df-4636-abbc-35fa6f8a164a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.782872 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-podinfo\") pod \"499b41f2-15df-4636-abbc-35fa6f8a164a\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.782902 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/499b41f2-15df-4636-abbc-35fa6f8a164a-kube-api-access-dm5hm\" (UniqueName: \"kubernetes.io/projected/499b41f2-15df-4636-abbc-35fa6f8a164a-kube-api-access-dm5hm\") pod \"499b41f2-15df-4636-abbc-35fa6f8a164a\" (UID: \"499b41f2-15df-4636-abbc-35fa6f8a164a\") " Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.783014 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-data" pod "199b78b1-0dcc-44df-a25b-9c401a86ad70" (UID: "199b78b1-0dcc-44df-a25b-9c401a86ad70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.783033 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-workload-certs\") pod \"199b78b1-0dcc-44df-a25b-9c401a86ad70\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.783077 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-workload-socket\") pod \"199b78b1-0dcc-44df-a25b-9c401a86ad70\" (UID: \"199b78b1-0dcc-44df-a25b-9c401a86ad70\") " Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.783187 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-credential-socket" pod "199b78b1-0dcc-44df-a25b-9c401a86ad70" (UID: "199b78b1-0dcc-44df-a25b-9c401a86ad70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.783429 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/499b41f2-15df-4636-abbc-35fa6f8a164a-istiod-ca-cert" pod "499b41f2-15df-4636-abbc-35fa6f8a164a" (UID: "499b41f2-15df-4636-abbc-35fa6f8a164a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.783604 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-workload-socket" pod "199b78b1-0dcc-44df-a25b-9c401a86ad70" (UID: "199b78b1-0dcc-44df-a25b-9c401a86ad70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.783663 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-credential-socket" pod "499b41f2-15df-4636-abbc-35fa6f8a164a" (UID: "499b41f2-15df-4636-abbc-35fa6f8a164a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.784201 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-workload-certs" pod "499b41f2-15df-4636-abbc-35fa6f8a164a" (UID: "499b41f2-15df-4636-abbc-35fa6f8a164a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.784270 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/199b78b1-0dcc-44df-a25b-9c401a86ad70-istiod-ca-cert" pod "199b78b1-0dcc-44df-a25b-9c401a86ad70" (UID: "199b78b1-0dcc-44df-a25b-9c401a86ad70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.785111 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/199b78b1-0dcc-44df-a25b-9c401a86ad70-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.785136 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/499b41f2-15df-4636-abbc-35fa6f8a164a-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.785150 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.785165 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.785178 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.785192 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.785205 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.785221 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:51.785675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.785234 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:51.792451 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.792276 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-workload-certs" pod "199b78b1-0dcc-44df-a25b-9c401a86ad70" (UID: "199b78b1-0dcc-44df-a25b-9c401a86ad70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:51.804653 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.804618 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-token" pod "199b78b1-0dcc-44df-a25b-9c401a86ad70" (UID: "199b78b1-0dcc-44df-a25b-9c401a86ad70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:50:51.808005 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.807980 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/499b41f2-15df-4636-abbc-35fa6f8a164a-kube-api-access-dm5hm" pod "499b41f2-15df-4636-abbc-35fa6f8a164a" (UID: "499b41f2-15df-4636-abbc-35fa6f8a164a"). InnerVolumeSpecName "kube-api-access-dm5hm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:50:51.819164 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.819111 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-podinfo" pod "199b78b1-0dcc-44df-a25b-9c401a86ad70" (UID: "199b78b1-0dcc-44df-a25b-9c401a86ad70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:50:51.824074 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.819350 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-podinfo" pod "499b41f2-15df-4636-abbc-35fa6f8a164a" (UID: "499b41f2-15df-4636-abbc-35fa6f8a164a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:50:51.824074 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.819383 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-token" pod "499b41f2-15df-4636-abbc-35fa6f8a164a" (UID: "499b41f2-15df-4636-abbc-35fa6f8a164a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:50:51.824074 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.819440 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-envoy" pod "199b78b1-0dcc-44df-a25b-9c401a86ad70" (UID: "199b78b1-0dcc-44df-a25b-9c401a86ad70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:51.824074 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.819484 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-envoy" pod "499b41f2-15df-4636-abbc-35fa6f8a164a" (UID: "499b41f2-15df-4636-abbc-35fa6f8a164a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:51.824074 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.819539 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/199b78b1-0dcc-44df-a25b-9c401a86ad70-kube-api-access-6rbb6" pod "199b78b1-0dcc-44df-a25b-9c401a86ad70" (UID: "199b78b1-0dcc-44df-a25b-9c401a86ad70"). InnerVolumeSpecName "kube-api-access-6rbb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:50:51.868359 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.868313 4910 patch_prober.go:29] interesting pod/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:50:51.868510 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.868381 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" podUID="706169ef-5dc5-4533-86af-fd02de456ab0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:50:51.887820 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.887786 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:51.887820 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.887823 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:51.887991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.887838 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:51.887991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.887853 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:51.887991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.887867 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm5hm\" (UniqueName: \"kubernetes.io/projected/499b41f2-15df-4636-abbc-35fa6f8a164a-kube-api-access-dm5hm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:51.887991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.887883 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/199b78b1-0dcc-44df-a25b-9c401a86ad70-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:51.887991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.887898 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/199b78b1-0dcc-44df-a25b-9c401a86ad70-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:51.887991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.887929 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/499b41f2-15df-4636-abbc-35fa6f8a164a-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:51.887991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:51.887943 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rbb6\" (UniqueName: \"kubernetes.io/projected/199b78b1-0dcc-44df-a25b-9c401a86ad70-kube-api-access-6rbb6\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:52.139925 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.139875 4910 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:50:52.140100 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.139955 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" podUID="be3bf6bf-4863-4489-b924-3de83b589c8e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:50:52.274257 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.274207 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85fa0a2-68tnp" event={"ID":"76ca271f-5d14-4807-928d-a4c5c23debc4","Type":"ContainerStarted","Data":"b9ccb137c1e6af7d2ebce38a00b1f3094b32cb2b861b0994b665aebfffc2c004"} Oct 31 19:50:52.274537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.274282 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85fa0a2-68tnp" event={"ID":"76ca271f-5d14-4807-928d-a4c5c23debc4","Type":"ContainerStarted","Data":"ff47ef101da9e840eb092b6803640bf46f3c2d8fd2a03fa2d241cbb7ff88dad2"} Oct 31 19:50:52.276986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.276953 4910 generic.go:356] "Generic (PLEG): container finished" podID="199b78b1-0dcc-44df-a25b-9c401a86ad70" containerID="64490afe30f522a499652851d571209250fcba5b54658150b7da616bfad14564" exitCode=0 Oct 31 19:50:52.277243 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.277004 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" Oct 31 19:50:52.277389 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.277019 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" event={"ID":"199b78b1-0dcc-44df-a25b-9c401a86ad70","Type":"ContainerDied","Data":"64490afe30f522a499652851d571209250fcba5b54658150b7da616bfad14564"} Oct 31 19:50:52.277389 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.277354 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch" event={"ID":"199b78b1-0dcc-44df-a25b-9c401a86ad70","Type":"ContainerDied","Data":"4b9cb99e587114ee41326bc96fb4fa6af84759be806b042114a65c0e58308e45"} Oct 31 19:50:52.277389 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.277386 4910 scope.go:122] "RemoveContainer" containerID="64490afe30f522a499652851d571209250fcba5b54658150b7da616bfad14564" Oct 31 19:50:52.280436 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.280416 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4" event={"ID":"01593536-a296-4333-a1a2-cca5cedeb8f9","Type":"ContainerDied","Data":"428a4a377c21a9855b0ede8af053398c59b0d969d687e3492294a0650247e4f4"} Oct 31 19:50:52.280692 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.280675 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4" Oct 31 19:50:52.287929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.287886 4910 generic.go:356] "Generic (PLEG): container finished" podID="499b41f2-15df-4636-abbc-35fa6f8a164a" containerID="853d31bcc99e84aa887bf15f15988fde746f71055e45e99b9091cf9f3f22322f" exitCode=0 Oct 31 19:50:52.288141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.288121 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" event={"ID":"499b41f2-15df-4636-abbc-35fa6f8a164a","Type":"ContainerDied","Data":"853d31bcc99e84aa887bf15f15988fde746f71055e45e99b9091cf9f3f22322f"} Oct 31 19:50:52.288220 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.288173 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" event={"ID":"499b41f2-15df-4636-abbc-35fa6f8a164a","Type":"ContainerDied","Data":"ce164ba30ac0c97c9d65abb7ef12f9bfeb5a9e1b318e8b2ab604b51cd61937c7"} Oct 31 19:50:52.288276 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.288250 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj" Oct 31 19:50:52.300223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.300173 4910 scope.go:122] "RemoveContainer" containerID="64490afe30f522a499652851d571209250fcba5b54658150b7da616bfad14564" Oct 31 19:50:52.301117 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:50:52.300537 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64490afe30f522a499652851d571209250fcba5b54658150b7da616bfad14564\": container with ID starting with 64490afe30f522a499652851d571209250fcba5b54658150b7da616bfad14564 not found: ID does not exist" containerID="64490afe30f522a499652851d571209250fcba5b54658150b7da616bfad14564" Oct 31 19:50:52.301117 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.300575 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64490afe30f522a499652851d571209250fcba5b54658150b7da616bfad14564"} err="failed to get container status \"64490afe30f522a499652851d571209250fcba5b54658150b7da616bfad14564\": rpc error: code = NotFound desc = could not find container \"64490afe30f522a499652851d571209250fcba5b54658150b7da616bfad14564\": container with ID starting with 64490afe30f522a499652851d571209250fcba5b54658150b7da616bfad14564 not found: ID does not exist" Oct 31 19:50:52.301117 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.300600 4910 scope.go:122] "RemoveContainer" containerID="b8adbe1c5e2ab4f141f43373af7c0affc7682f476e812bff205d349731ad9ca4" Oct 31 19:50:52.316115 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.316089 4910 scope.go:122] "RemoveContainer" containerID="853d31bcc99e84aa887bf15f15988fde746f71055e45e99b9091cf9f3f22322f" Oct 31 19:50:52.332354 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.332327 4910 scope.go:122] "RemoveContainer" containerID="ee5138499d94a28ba9e5fff1962310587ee29c357947bdfd6e3be726302de64c" Oct 31 19:50:52.334387 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.334336 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85fa0a2-68tnp" podStartSLOduration=3.334317529 podStartE2EDuration="3.334317529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:50:52.331969363 +0000 UTC m=+1886.694789736" watchObservedRunningTime="2026-10-31 19:50:52.334317529 +0000 UTC m=+1886.697137902" Oct 31 19:50:52.369844 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.369806 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4"] Oct 31 19:50:52.370114 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.369974 4910 scope.go:122] "RemoveContainer" containerID="853d31bcc99e84aa887bf15f15988fde746f71055e45e99b9091cf9f3f22322f" Oct 31 19:50:52.370475 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:50:52.370442 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"853d31bcc99e84aa887bf15f15988fde746f71055e45e99b9091cf9f3f22322f\": container with ID starting with 853d31bcc99e84aa887bf15f15988fde746f71055e45e99b9091cf9f3f22322f not found: ID does not exist" containerID="853d31bcc99e84aa887bf15f15988fde746f71055e45e99b9091cf9f3f22322f" Oct 31 19:50:52.370578 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.370484 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"853d31bcc99e84aa887bf15f15988fde746f71055e45e99b9091cf9f3f22322f"} err="failed to get container status \"853d31bcc99e84aa887bf15f15988fde746f71055e45e99b9091cf9f3f22322f\": rpc error: code = NotFound desc = could not find container \"853d31bcc99e84aa887bf15f15988fde746f71055e45e99b9091cf9f3f22322f\": container with ID starting with 853d31bcc99e84aa887bf15f15988fde746f71055e45e99b9091cf9f3f22322f not found: ID does not exist" Oct 31 19:50:52.370578 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.370510 4910 scope.go:122] "RemoveContainer" containerID="ee5138499d94a28ba9e5fff1962310587ee29c357947bdfd6e3be726302de64c" Oct 31 19:50:52.371400 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:50:52.371372 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee5138499d94a28ba9e5fff1962310587ee29c357947bdfd6e3be726302de64c\": container with ID starting with ee5138499d94a28ba9e5fff1962310587ee29c357947bdfd6e3be726302de64c not found: ID does not exist" containerID="ee5138499d94a28ba9e5fff1962310587ee29c357947bdfd6e3be726302de64c" Oct 31 19:50:52.371508 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.371409 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee5138499d94a28ba9e5fff1962310587ee29c357947bdfd6e3be726302de64c"} err="failed to get container status \"ee5138499d94a28ba9e5fff1962310587ee29c357947bdfd6e3be726302de64c\": rpc error: code = NotFound desc = could not find container \"ee5138499d94a28ba9e5fff1962310587ee29c357947bdfd6e3be726302de64c\": container with ID starting with ee5138499d94a28ba9e5fff1962310587ee29c357947bdfd6e3be726302de64c not found: ID does not exist" Oct 31 19:50:52.375951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.375928 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a4e312-868b4d9d46-z7zb4"] Oct 31 19:50:52.424388 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.424258 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch"] Oct 31 19:50:52.434664 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.434593 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-55lch"] Oct 31 19:50:52.526001 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.525964 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj"] Oct 31 19:50:52.535271 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.535240 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-q68jj"] Oct 31 19:50:52.760780 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.760692 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq"] Oct 31 19:50:52.761321 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:52.761285 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" podUID="9f646f52-3c80-4fbb-8049-712f1e6b1595" containerName="istio-proxy" containerID="cri-o://49b296095d99cebbcb1cf079e727995b87590d86300fbcf6e1c33012c4842652" gracePeriod=30 Oct 31 19:50:53.076060 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.076014 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j" podUID="09ee3f2a-332e-4c98-9b21-c76f3903b69f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:50:53.323117 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.323083 4910 generic.go:356] "Generic (PLEG): container finished" podID="76ca271f-5d14-4807-928d-a4c5c23debc4" containerID="b9ccb137c1e6af7d2ebce38a00b1f3094b32cb2b861b0994b665aebfffc2c004" exitCode=0 Oct 31 19:50:53.323325 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.323164 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85fa0a2-68tnp" event={"ID":"76ca271f-5d14-4807-928d-a4c5c23debc4","Type":"ContainerDied","Data":"b9ccb137c1e6af7d2ebce38a00b1f3094b32cb2b861b0994b665aebfffc2c004"} Oct 31 19:50:53.329137 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.329064 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft"] Oct 31 19:50:53.330346 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.330316 4910 cpu_manager.go:414] "containerMap: removing container" podUID="499b41f2-15df-4636-abbc-35fa6f8a164a" containerName="istio-proxy" Oct 31 19:50:53.330346 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.330344 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="499b41f2-15df-4636-abbc-35fa6f8a164a" containerName="istio-proxy" Oct 31 19:50:53.330529 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.330388 4910 cpu_manager.go:414] "containerMap: removing container" podUID="199b78b1-0dcc-44df-a25b-9c401a86ad70" containerName="istio-proxy" Oct 31 19:50:53.330529 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.330396 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="199b78b1-0dcc-44df-a25b-9c401a86ad70" containerName="istio-proxy" Oct 31 19:50:53.330529 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.330409 4910 cpu_manager.go:414] "containerMap: removing container" podUID="01593536-a296-4333-a1a2-cca5cedeb8f9" containerName="maas-api" Oct 31 19:50:53.330529 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.330416 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01593536-a296-4333-a1a2-cca5cedeb8f9" containerName="maas-api" Oct 31 19:50:53.330529 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.330464 4910 cpu_manager.go:414] "containerMap: removing container" podUID="b97ba48f-5c91-4007-bb09-a54f35264beb" containerName="revoke-keys" Oct 31 19:50:53.330529 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.330472 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b97ba48f-5c91-4007-bb09-a54f35264beb" containerName="revoke-keys" Oct 31 19:50:53.330529 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.330486 4910 cpu_manager.go:414] "containerMap: removing container" podUID="499b41f2-15df-4636-abbc-35fa6f8a164a" containerName="istio-proxy" Oct 31 19:50:53.330529 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.330493 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="499b41f2-15df-4636-abbc-35fa6f8a164a" containerName="istio-proxy" Oct 31 19:50:53.331035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.330702 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="499b41f2-15df-4636-abbc-35fa6f8a164a" containerName="istio-proxy" Oct 31 19:50:53.331035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.330722 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="199b78b1-0dcc-44df-a25b-9c401a86ad70" containerName="istio-proxy" Oct 31 19:50:53.331035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.330741 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="01593536-a296-4333-a1a2-cca5cedeb8f9" containerName="maas-api" Oct 31 19:50:53.331035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.330756 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="b97ba48f-5c91-4007-bb09-a54f35264beb" containerName="revoke-keys" Oct 31 19:50:53.335501 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.335480 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft" Oct 31 19:50:53.366010 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.365979 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft"] Oct 31 19:50:53.455674 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.455637 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5928c0b9-c985-4b45-bd5e-b0543e262336-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft\" (UID: \"5928c0b9-c985-4b45-bd5e-b0543e262336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft" Oct 31 19:50:53.455899 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.455703 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5928c0b9-c985-4b45-bd5e-b0543e262336-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft\" (UID: \"5928c0b9-c985-4b45-bd5e-b0543e262336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft" Oct 31 19:50:53.455899 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.455881 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w76z\" (UniqueName: \"kubernetes.io/projected/5928c0b9-c985-4b45-bd5e-b0543e262336-kube-api-access-5w76z\") pod \"maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft\" (UID: \"5928c0b9-c985-4b45-bd5e-b0543e262336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft" Oct 31 19:50:53.558032 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.557994 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5928c0b9-c985-4b45-bd5e-b0543e262336-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft\" (UID: \"5928c0b9-c985-4b45-bd5e-b0543e262336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft" Oct 31 19:50:53.558557 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.558052 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5928c0b9-c985-4b45-bd5e-b0543e262336-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft\" (UID: \"5928c0b9-c985-4b45-bd5e-b0543e262336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft" Oct 31 19:50:53.558557 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.558107 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w76z\" (UniqueName: \"kubernetes.io/projected/5928c0b9-c985-4b45-bd5e-b0543e262336-kube-api-access-5w76z\") pod \"maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft\" (UID: \"5928c0b9-c985-4b45-bd5e-b0543e262336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft" Oct 31 19:50:53.558973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.558945 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5928c0b9-c985-4b45-bd5e-b0543e262336-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft\" (UID: \"5928c0b9-c985-4b45-bd5e-b0543e262336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft" Oct 31 19:50:53.560623 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.560598 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5928c0b9-c985-4b45-bd5e-b0543e262336-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft\" (UID: \"5928c0b9-c985-4b45-bd5e-b0543e262336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft" Oct 31 19:50:53.576381 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.576312 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w76z\" (UniqueName: \"kubernetes.io/projected/5928c0b9-c985-4b45-bd5e-b0543e262336-kube-api-access-5w76z\") pod \"maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft\" (UID: \"5928c0b9-c985-4b45-bd5e-b0543e262336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft" Oct 31 19:50:53.654549 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.654451 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft" Oct 31 19:50:53.788363 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.788324 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01593536-a296-4333-a1a2-cca5cedeb8f9" path="/var/lib/kubelet/pods/01593536-a296-4333-a1a2-cca5cedeb8f9/volumes" Oct 31 19:50:53.789353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.789316 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="199b78b1-0dcc-44df-a25b-9c401a86ad70" path="/var/lib/kubelet/pods/199b78b1-0dcc-44df-a25b-9c401a86ad70/volumes" Oct 31 19:50:53.790190 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.790169 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="499b41f2-15df-4636-abbc-35fa6f8a164a" path="/var/lib/kubelet/pods/499b41f2-15df-4636-abbc-35fa6f8a164a/volumes" Oct 31 19:50:53.974322 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:53.974292 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft"] Oct 31 19:50:54.335814 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.335785 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft" event={"ID":"5928c0b9-c985-4b45-bd5e-b0543e262336","Type":"ContainerStarted","Data":"1bbdcfc17cb3b535ef19bda1c38f4c0e2995828950b63dc6c5b81b3c3308fb91"} Oct 31 19:50:54.520518 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.520435 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85fa0a2-68tnp" Oct 31 19:50:54.608660 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.608618 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ca271f-5d14-4807-928d-a4c5c23debc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ca271f-5d14-4807-928d-a4c5c23debc4-maas-api-openshift-ca\") pod \"76ca271f-5d14-4807-928d-a4c5c23debc4\" (UID: \"76ca271f-5d14-4807-928d-a4c5c23debc4\") " Oct 31 19:50:54.609194 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.608769 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76ca271f-5d14-4807-928d-a4c5c23debc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ca271f-5d14-4807-928d-a4c5c23debc4-maas-api-xks-ca\") pod \"76ca271f-5d14-4807-928d-a4c5c23debc4\" (UID: \"76ca271f-5d14-4807-928d-a4c5c23debc4\") " Oct 31 19:50:54.609194 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.608896 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ca271f-5d14-4807-928d-a4c5c23debc4-kube-api-access-9lwwm\" (UniqueName: \"kubernetes.io/projected/76ca271f-5d14-4807-928d-a4c5c23debc4-kube-api-access-9lwwm\") pod \"76ca271f-5d14-4807-928d-a4c5c23debc4\" (UID: \"76ca271f-5d14-4807-928d-a4c5c23debc4\") " Oct 31 19:50:54.612935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.611300 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ca271f-5d14-4807-928d-a4c5c23debc4-maas-api-openshift-ca" pod "76ca271f-5d14-4807-928d-a4c5c23debc4" (UID: "76ca271f-5d14-4807-928d-a4c5c23debc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:50:54.614016 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.613873 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76ca271f-5d14-4807-928d-a4c5c23debc4-maas-api-xks-ca" pod "76ca271f-5d14-4807-928d-a4c5c23debc4" (UID: "76ca271f-5d14-4807-928d-a4c5c23debc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:50:54.629996 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.626317 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ca271f-5d14-4807-928d-a4c5c23debc4-kube-api-access-9lwwm" pod "76ca271f-5d14-4807-928d-a4c5c23debc4" (UID: "76ca271f-5d14-4807-928d-a4c5c23debc4"). InnerVolumeSpecName "kube-api-access-9lwwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:50:54.709317 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.709288 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:50:54.711950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.711860 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ca271f-5d14-4807-928d-a4c5c23debc4-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:54.711950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.711894 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ca271f-5d14-4807-928d-a4c5c23debc4-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:54.711950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.711926 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lwwm\" (UniqueName: \"kubernetes.io/projected/76ca271f-5d14-4807-928d-a4c5c23debc4-kube-api-access-9lwwm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:54.813232 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.813196 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-credential-socket\") pod \"706169ef-5dc5-4533-86af-fd02de456ab0\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " Oct 31 19:50:54.813429 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.813257 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/706169ef-5dc5-4533-86af-fd02de456ab0-kube-api-access-5dv5k\" (UniqueName: \"kubernetes.io/projected/706169ef-5dc5-4533-86af-fd02de456ab0-kube-api-access-5dv5k\") pod \"706169ef-5dc5-4533-86af-fd02de456ab0\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " Oct 31 19:50:54.813429 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.813343 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-istio-data\") pod \"706169ef-5dc5-4533-86af-fd02de456ab0\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " Oct 31 19:50:54.813578 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.813552 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-workload-socket\") pod \"706169ef-5dc5-4533-86af-fd02de456ab0\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " Oct 31 19:50:54.813680 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.813662 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-istio-envoy\") pod \"706169ef-5dc5-4533-86af-fd02de456ab0\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " Oct 31 19:50:54.813726 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.813700 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-workload-certs\") pod \"706169ef-5dc5-4533-86af-fd02de456ab0\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " Oct 31 19:50:54.813821 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.813801 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/706169ef-5dc5-4533-86af-fd02de456ab0-istio-token\" (UniqueName: \"kubernetes.io/projected/706169ef-5dc5-4533-86af-fd02de456ab0-istio-token\") pod \"706169ef-5dc5-4533-86af-fd02de456ab0\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " Oct 31 19:50:54.813889 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.813839 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-credential-socket" pod "706169ef-5dc5-4533-86af-fd02de456ab0" (UID: "706169ef-5dc5-4533-86af-fd02de456ab0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:54.813889 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.813851 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/706169ef-5dc5-4533-86af-fd02de456ab0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/706169ef-5dc5-4533-86af-fd02de456ab0-istiod-ca-cert\") pod \"706169ef-5dc5-4533-86af-fd02de456ab0\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " Oct 31 19:50:54.814022 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.813896 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/706169ef-5dc5-4533-86af-fd02de456ab0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/706169ef-5dc5-4533-86af-fd02de456ab0-istio-podinfo\") pod \"706169ef-5dc5-4533-86af-fd02de456ab0\" (UID: \"706169ef-5dc5-4533-86af-fd02de456ab0\") " Oct 31 19:50:54.814084 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.814018 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-workload-socket" pod "706169ef-5dc5-4533-86af-fd02de456ab0" (UID: "706169ef-5dc5-4533-86af-fd02de456ab0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:54.814150 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.814098 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-workload-certs" pod "706169ef-5dc5-4533-86af-fd02de456ab0" (UID: "706169ef-5dc5-4533-86af-fd02de456ab0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:54.815011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.814547 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-istio-data" pod "706169ef-5dc5-4533-86af-fd02de456ab0" (UID: "706169ef-5dc5-4533-86af-fd02de456ab0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:54.815011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.814968 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:54.815011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.814995 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:54.815011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.815009 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:54.815290 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.815022 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:54.815734 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.815702 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/706169ef-5dc5-4533-86af-fd02de456ab0-istiod-ca-cert" pod "706169ef-5dc5-4533-86af-fd02de456ab0" (UID: "706169ef-5dc5-4533-86af-fd02de456ab0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:50:54.816975 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.816906 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/706169ef-5dc5-4533-86af-fd02de456ab0-istio-token" pod "706169ef-5dc5-4533-86af-fd02de456ab0" (UID: "706169ef-5dc5-4533-86af-fd02de456ab0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:50:54.817271 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.817242 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/706169ef-5dc5-4533-86af-fd02de456ab0-istio-podinfo" pod "706169ef-5dc5-4533-86af-fd02de456ab0" (UID: "706169ef-5dc5-4533-86af-fd02de456ab0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:50:54.829573 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.829495 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/706169ef-5dc5-4533-86af-fd02de456ab0-kube-api-access-5dv5k" pod "706169ef-5dc5-4533-86af-fd02de456ab0" (UID: "706169ef-5dc5-4533-86af-fd02de456ab0"). InnerVolumeSpecName "kube-api-access-5dv5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:50:54.831280 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.831254 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-istio-envoy" pod "706169ef-5dc5-4533-86af-fd02de456ab0" (UID: "706169ef-5dc5-4533-86af-fd02de456ab0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:54.916755 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.916725 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/706169ef-5dc5-4533-86af-fd02de456ab0-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:54.916755 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.916752 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/706169ef-5dc5-4533-86af-fd02de456ab0-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:54.916755 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.916762 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/706169ef-5dc5-4533-86af-fd02de456ab0-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:54.917055 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.916774 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/706169ef-5dc5-4533-86af-fd02de456ab0-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:54.917055 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:54.916783 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dv5k\" (UniqueName: \"kubernetes.io/projected/706169ef-5dc5-4533-86af-fd02de456ab0-kube-api-access-5dv5k\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:55.346329 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:55.346298 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85fa0a2-68tnp" Oct 31 19:50:55.346694 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:55.346661 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85fa0a2-68tnp" event={"ID":"76ca271f-5d14-4807-928d-a4c5c23debc4","Type":"ContainerDied","Data":"ff47ef101da9e840eb092b6803640bf46f3c2d8fd2a03fa2d241cbb7ff88dad2"} Oct 31 19:50:55.346760 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:55.346724 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff47ef101da9e840eb092b6803640bf46f3c2d8fd2a03fa2d241cbb7ff88dad2" Oct 31 19:50:55.348282 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:55.348252 4910 generic.go:356] "Generic (PLEG): container finished" podID="5928c0b9-c985-4b45-bd5e-b0543e262336" containerID="a16125cd51034e99dc1721001f74d87ca55f46e79dd1cbab3452162c339e9297" exitCode=0 Oct 31 19:50:55.348394 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:55.348338 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft" event={"ID":"5928c0b9-c985-4b45-bd5e-b0543e262336","Type":"ContainerDied","Data":"a16125cd51034e99dc1721001f74d87ca55f46e79dd1cbab3452162c339e9297"} Oct 31 19:50:55.350467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:55.350443 4910 generic.go:356] "Generic (PLEG): container finished" podID="706169ef-5dc5-4533-86af-fd02de456ab0" containerID="cc37f5426fae5d07363bc461faf8a4d307bc71f4bdb50f30f785a298622423b0" exitCode=0 Oct 31 19:50:55.350584 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:55.350493 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" Oct 31 19:50:55.350584 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:55.350495 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" event={"ID":"706169ef-5dc5-4533-86af-fd02de456ab0","Type":"ContainerDied","Data":"cc37f5426fae5d07363bc461faf8a4d307bc71f4bdb50f30f785a298622423b0"} Oct 31 19:50:55.350686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:55.350607 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd" event={"ID":"706169ef-5dc5-4533-86af-fd02de456ab0","Type":"ContainerDied","Data":"3bc4c4891a05c3274c252f9a3eaad890273a632dcf6a7ff687b28b922ed363f4"} Oct 31 19:50:55.350686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:55.350645 4910 scope.go:122] "RemoveContainer" containerID="cc37f5426fae5d07363bc461faf8a4d307bc71f4bdb50f30f785a298622423b0" Oct 31 19:50:55.365267 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:55.365249 4910 scope.go:122] "RemoveContainer" containerID="c88f574089b8579100c712a86a00aa52ee8ae7808ac21f7ef64c261a98cbcff8" Oct 31 19:50:55.376768 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:55.376747 4910 scope.go:122] "RemoveContainer" containerID="cc37f5426fae5d07363bc461faf8a4d307bc71f4bdb50f30f785a298622423b0" Oct 31 19:50:55.377121 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:50:55.377096 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc37f5426fae5d07363bc461faf8a4d307bc71f4bdb50f30f785a298622423b0\": container with ID starting with cc37f5426fae5d07363bc461faf8a4d307bc71f4bdb50f30f785a298622423b0 not found: ID does not exist" containerID="cc37f5426fae5d07363bc461faf8a4d307bc71f4bdb50f30f785a298622423b0" Oct 31 19:50:55.377178 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:55.377131 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc37f5426fae5d07363bc461faf8a4d307bc71f4bdb50f30f785a298622423b0"} err="failed to get container status \"cc37f5426fae5d07363bc461faf8a4d307bc71f4bdb50f30f785a298622423b0\": rpc error: code = NotFound desc = could not find container \"cc37f5426fae5d07363bc461faf8a4d307bc71f4bdb50f30f785a298622423b0\": container with ID starting with cc37f5426fae5d07363bc461faf8a4d307bc71f4bdb50f30f785a298622423b0 not found: ID does not exist" Oct 31 19:50:55.377178 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:55.377149 4910 scope.go:122] "RemoveContainer" containerID="c88f574089b8579100c712a86a00aa52ee8ae7808ac21f7ef64c261a98cbcff8" Oct 31 19:50:55.377508 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:50:55.377481 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c88f574089b8579100c712a86a00aa52ee8ae7808ac21f7ef64c261a98cbcff8\": container with ID starting with c88f574089b8579100c712a86a00aa52ee8ae7808ac21f7ef64c261a98cbcff8 not found: ID does not exist" containerID="c88f574089b8579100c712a86a00aa52ee8ae7808ac21f7ef64c261a98cbcff8" Oct 31 19:50:55.377590 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:55.377515 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c88f574089b8579100c712a86a00aa52ee8ae7808ac21f7ef64c261a98cbcff8"} err="failed to get container status \"c88f574089b8579100c712a86a00aa52ee8ae7808ac21f7ef64c261a98cbcff8\": rpc error: code = NotFound desc = could not find container \"c88f574089b8579100c712a86a00aa52ee8ae7808ac21f7ef64c261a98cbcff8\": container with ID starting with c88f574089b8579100c712a86a00aa52ee8ae7808ac21f7ef64c261a98cbcff8 not found: ID does not exist" Oct 31 19:50:55.407263 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:55.407230 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd"] Oct 31 19:50:55.416096 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:55.416063 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-sp7bd"] Oct 31 19:50:55.761618 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:55.761536 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="706169ef-5dc5-4533-86af-fd02de456ab0" path="/var/lib/kubelet/pods/706169ef-5dc5-4533-86af-fd02de456ab0/volumes" Oct 31 19:50:56.739125 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:56.738557 4910 generic.go:356] "Generic (PLEG): container finished" podID="be3bf6bf-4863-4489-b924-3de83b589c8e" containerID="eb4d5dc8612c55538b4fe5a87db69d023da6d807ad5f4617dbbf7d0858497bb3" exitCode=0 Oct 31 19:50:56.739125 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:56.738766 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" event={"ID":"be3bf6bf-4863-4489-b924-3de83b589c8e","Type":"ContainerDied","Data":"eb4d5dc8612c55538b4fe5a87db69d023da6d807ad5f4617dbbf7d0858497bb3"} Oct 31 19:50:56.948396 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:56.948366 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:50:57.193937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.188677 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be3bf6bf-4863-4489-b924-3de83b589c8e-kube-api-access-pw28n\" (UniqueName: \"kubernetes.io/projected/be3bf6bf-4863-4489-b924-3de83b589c8e-kube-api-access-pw28n\") pod \"be3bf6bf-4863-4489-b924-3de83b589c8e\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " Oct 31 19:50:57.193937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.188820 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be3bf6bf-4863-4489-b924-3de83b589c8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be3bf6bf-4863-4489-b924-3de83b589c8e-istiod-ca-cert\") pod \"be3bf6bf-4863-4489-b924-3de83b589c8e\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " Oct 31 19:50:57.193937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.189008 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-envoy\") pod \"be3bf6bf-4863-4489-b924-3de83b589c8e\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " Oct 31 19:50:57.193937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.189044 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-token\" (UniqueName: \"kubernetes.io/projected/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-token\") pod \"be3bf6bf-4863-4489-b924-3de83b589c8e\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " Oct 31 19:50:57.193937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.189081 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-workload-certs\") pod \"be3bf6bf-4863-4489-b924-3de83b589c8e\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " Oct 31 19:50:57.193937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.189214 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-podinfo\") pod \"be3bf6bf-4863-4489-b924-3de83b589c8e\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " Oct 31 19:50:57.193937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.189247 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-workload-socket\") pod \"be3bf6bf-4863-4489-b924-3de83b589c8e\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " Oct 31 19:50:57.193937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.189294 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-credential-socket\") pod \"be3bf6bf-4863-4489-b924-3de83b589c8e\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " Oct 31 19:50:57.193937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.189343 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-data\") pod \"be3bf6bf-4863-4489-b924-3de83b589c8e\" (UID: \"be3bf6bf-4863-4489-b924-3de83b589c8e\") " Oct 31 19:50:57.193937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.190649 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-workload-certs" pod "be3bf6bf-4863-4489-b924-3de83b589c8e" (UID: "be3bf6bf-4863-4489-b924-3de83b589c8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:57.217606 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.199032 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-workload-socket" pod "be3bf6bf-4863-4489-b924-3de83b589c8e" (UID: "be3bf6bf-4863-4489-b924-3de83b589c8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:57.217606 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.214192 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be3bf6bf-4863-4489-b924-3de83b589c8e-istiod-ca-cert" pod "be3bf6bf-4863-4489-b924-3de83b589c8e" (UID: "be3bf6bf-4863-4489-b924-3de83b589c8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:50:57.232936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.219461 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-data" pod "be3bf6bf-4863-4489-b924-3de83b589c8e" (UID: "be3bf6bf-4863-4489-b924-3de83b589c8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:57.250513 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.234013 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-credential-socket" pod "be3bf6bf-4863-4489-b924-3de83b589c8e" (UID: "be3bf6bf-4863-4489-b924-3de83b589c8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:57.250513 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.239947 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-podinfo" pod "be3bf6bf-4863-4489-b924-3de83b589c8e" (UID: "be3bf6bf-4863-4489-b924-3de83b589c8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:50:57.250513 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.240077 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-envoy" pod "be3bf6bf-4863-4489-b924-3de83b589c8e" (UID: "be3bf6bf-4863-4489-b924-3de83b589c8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:57.250513 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.240431 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-token" pod "be3bf6bf-4863-4489-b924-3de83b589c8e" (UID: "be3bf6bf-4863-4489-b924-3de83b589c8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:50:57.250513 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.242780 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be3bf6bf-4863-4489-b924-3de83b589c8e-kube-api-access-pw28n" pod "be3bf6bf-4863-4489-b924-3de83b589c8e" (UID: "be3bf6bf-4863-4489-b924-3de83b589c8e"). InnerVolumeSpecName "kube-api-access-pw28n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:50:57.294281 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.292821 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:57.294281 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.292865 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:57.294281 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.292883 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:57.371081 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.360092 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:57.371081 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.360144 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:57.371081 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.360164 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:57.371081 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.360180 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be3bf6bf-4863-4489-b924-3de83b589c8e-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:57.371081 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.360206 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw28n\" (UniqueName: \"kubernetes.io/projected/be3bf6bf-4863-4489-b924-3de83b589c8e-kube-api-access-pw28n\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:57.371081 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.360231 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be3bf6bf-4863-4489-b924-3de83b589c8e-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:57.406856 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.406827 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft" Oct 31 19:50:57.562138 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.562105 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5928c0b9-c985-4b45-bd5e-b0543e262336-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5928c0b9-c985-4b45-bd5e-b0543e262336-maas-api-xks-ca\") pod \"5928c0b9-c985-4b45-bd5e-b0543e262336\" (UID: \"5928c0b9-c985-4b45-bd5e-b0543e262336\") " Oct 31 19:50:57.562342 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.562145 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5928c0b9-c985-4b45-bd5e-b0543e262336-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5928c0b9-c985-4b45-bd5e-b0543e262336-maas-api-openshift-ca\") pod \"5928c0b9-c985-4b45-bd5e-b0543e262336\" (UID: \"5928c0b9-c985-4b45-bd5e-b0543e262336\") " Oct 31 19:50:57.562442 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.562413 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5928c0b9-c985-4b45-bd5e-b0543e262336-kube-api-access-5w76z\" (UniqueName: \"kubernetes.io/projected/5928c0b9-c985-4b45-bd5e-b0543e262336-kube-api-access-5w76z\") pod \"5928c0b9-c985-4b45-bd5e-b0543e262336\" (UID: \"5928c0b9-c985-4b45-bd5e-b0543e262336\") " Oct 31 19:50:57.562589 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.562568 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5928c0b9-c985-4b45-bd5e-b0543e262336-maas-api-openshift-ca" pod "5928c0b9-c985-4b45-bd5e-b0543e262336" (UID: "5928c0b9-c985-4b45-bd5e-b0543e262336"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:50:57.563110 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.563092 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5928c0b9-c985-4b45-bd5e-b0543e262336-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:57.564266 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.564242 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5928c0b9-c985-4b45-bd5e-b0543e262336-maas-api-xks-ca" pod "5928c0b9-c985-4b45-bd5e-b0543e262336" (UID: "5928c0b9-c985-4b45-bd5e-b0543e262336"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:50:57.564385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.564363 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5928c0b9-c985-4b45-bd5e-b0543e262336-kube-api-access-5w76z" pod "5928c0b9-c985-4b45-bd5e-b0543e262336" (UID: "5928c0b9-c985-4b45-bd5e-b0543e262336"). InnerVolumeSpecName "kube-api-access-5w76z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:50:57.664462 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.664429 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5928c0b9-c985-4b45-bd5e-b0543e262336-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:57.664462 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.664461 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w76z\" (UniqueName: \"kubernetes.io/projected/5928c0b9-c985-4b45-bd5e-b0543e262336-kube-api-access-5w76z\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:57.776625 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.776588 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft" event={"ID":"5928c0b9-c985-4b45-bd5e-b0543e262336","Type":"ContainerDied","Data":"1bbdcfc17cb3b535ef19bda1c38f4c0e2995828950b63dc6c5b81b3c3308fb91"} Oct 31 19:50:57.776839 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.776647 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bbdcfc17cb3b535ef19bda1c38f4c0e2995828950b63dc6c5b81b3c3308fb91" Oct 31 19:50:57.776839 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.776602 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft" Oct 31 19:50:57.780136 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.779726 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" Oct 31 19:50:57.780136 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.779973 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w" event={"ID":"be3bf6bf-4863-4489-b924-3de83b589c8e","Type":"ContainerDied","Data":"ad552c55fefc09844acf4440ad2a912915e6c083570317a80245046124cd44df"} Oct 31 19:50:57.780136 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.780035 4910 scope.go:122] "RemoveContainer" containerID="eb4d5dc8612c55538b4fe5a87db69d023da6d807ad5f4617dbbf7d0858497bb3" Oct 31 19:50:57.835018 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.834932 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w"] Oct 31 19:50:57.842692 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:57.842662 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-g7x6w"] Oct 31 19:50:58.070045 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.070021 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:50:58.173284 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.173187 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f646f52-3c80-4fbb-8049-712f1e6b1595-kube-api-access-nhm4w\" (UniqueName: \"kubernetes.io/projected/9f646f52-3c80-4fbb-8049-712f1e6b1595-kube-api-access-nhm4w\") pod \"9f646f52-3c80-4fbb-8049-712f1e6b1595\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " Oct 31 19:50:58.173284 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.173243 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-envoy\") pod \"9f646f52-3c80-4fbb-8049-712f1e6b1595\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " Oct 31 19:50:58.173609 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.173307 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-podinfo\") pod \"9f646f52-3c80-4fbb-8049-712f1e6b1595\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " Oct 31 19:50:58.173609 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.173498 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-workload-certs\") pod \"9f646f52-3c80-4fbb-8049-712f1e6b1595\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " Oct 31 19:50:58.173609 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.173537 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-token\" (UniqueName: \"kubernetes.io/projected/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-token\") pod \"9f646f52-3c80-4fbb-8049-712f1e6b1595\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " Oct 31 19:50:58.173792 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.173634 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f646f52-3c80-4fbb-8049-712f1e6b1595-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f646f52-3c80-4fbb-8049-712f1e6b1595-istiod-ca-cert\") pod \"9f646f52-3c80-4fbb-8049-712f1e6b1595\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " Oct 31 19:50:58.173792 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.173675 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-data\") pod \"9f646f52-3c80-4fbb-8049-712f1e6b1595\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " Oct 31 19:50:58.173792 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.173767 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-workload-socket\") pod \"9f646f52-3c80-4fbb-8049-712f1e6b1595\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " Oct 31 19:50:58.173996 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.173874 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-credential-socket\") pod \"9f646f52-3c80-4fbb-8049-712f1e6b1595\" (UID: \"9f646f52-3c80-4fbb-8049-712f1e6b1595\") " Oct 31 19:50:58.174061 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.174031 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-workload-certs" pod "9f646f52-3c80-4fbb-8049-712f1e6b1595" (UID: "9f646f52-3c80-4fbb-8049-712f1e6b1595"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:58.175372 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.174322 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f646f52-3c80-4fbb-8049-712f1e6b1595-istiod-ca-cert" pod "9f646f52-3c80-4fbb-8049-712f1e6b1595" (UID: "9f646f52-3c80-4fbb-8049-712f1e6b1595"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:50:58.175372 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.174408 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-data" pod "9f646f52-3c80-4fbb-8049-712f1e6b1595" (UID: "9f646f52-3c80-4fbb-8049-712f1e6b1595"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:58.175372 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.174580 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-credential-socket" pod "9f646f52-3c80-4fbb-8049-712f1e6b1595" (UID: "9f646f52-3c80-4fbb-8049-712f1e6b1595"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:58.175372 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.174748 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-workload-socket" pod "9f646f52-3c80-4fbb-8049-712f1e6b1595" (UID: "9f646f52-3c80-4fbb-8049-712f1e6b1595"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:58.175372 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.175155 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:58.175372 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.175180 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:58.175372 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.175197 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:58.175372 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.175210 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f646f52-3c80-4fbb-8049-712f1e6b1595-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:58.175372 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.175224 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:58.176232 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.176203 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-envoy" pod "9f646f52-3c80-4fbb-8049-712f1e6b1595" (UID: "9f646f52-3c80-4fbb-8049-712f1e6b1595"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:50:58.176652 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.176618 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f646f52-3c80-4fbb-8049-712f1e6b1595-kube-api-access-nhm4w" pod "9f646f52-3c80-4fbb-8049-712f1e6b1595" (UID: "9f646f52-3c80-4fbb-8049-712f1e6b1595"). InnerVolumeSpecName "kube-api-access-nhm4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:50:58.177528 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.177502 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-podinfo" pod "9f646f52-3c80-4fbb-8049-712f1e6b1595" (UID: "9f646f52-3c80-4fbb-8049-712f1e6b1595"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:50:58.177616 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.177538 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-token" pod "9f646f52-3c80-4fbb-8049-712f1e6b1595" (UID: "9f646f52-3c80-4fbb-8049-712f1e6b1595"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:50:58.276967 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.276891 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:58.276967 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.276957 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhm4w\" (UniqueName: \"kubernetes.io/projected/9f646f52-3c80-4fbb-8049-712f1e6b1595-kube-api-access-nhm4w\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:58.276967 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.276967 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:58.276967 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.276977 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f646f52-3c80-4fbb-8049-712f1e6b1595-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:50:58.792536 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.792454 4910 generic.go:356] "Generic (PLEG): container finished" podID="9f646f52-3c80-4fbb-8049-712f1e6b1595" containerID="49b296095d99cebbcb1cf079e727995b87590d86300fbcf6e1c33012c4842652" exitCode=0 Oct 31 19:50:58.792536 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.792509 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" Oct 31 19:50:58.792790 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.792517 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" event={"ID":"9f646f52-3c80-4fbb-8049-712f1e6b1595","Type":"ContainerDied","Data":"49b296095d99cebbcb1cf079e727995b87590d86300fbcf6e1c33012c4842652"} Oct 31 19:50:58.792790 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.792649 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq" event={"ID":"9f646f52-3c80-4fbb-8049-712f1e6b1595","Type":"ContainerDied","Data":"41121d7f2e10a55856058b531afc3c956cd5d5aca34faef975214bc6f0b29cb7"} Oct 31 19:50:58.792790 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.792679 4910 scope.go:122] "RemoveContainer" containerID="49b296095d99cebbcb1cf079e727995b87590d86300fbcf6e1c33012c4842652" Oct 31 19:50:58.805768 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.805747 4910 scope.go:122] "RemoveContainer" containerID="6f3b83ca6f6ad605de24d7751c2606b03ebbe6624508c6c6865d37a6e9523cbb" Oct 31 19:50:58.816618 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.816600 4910 scope.go:122] "RemoveContainer" containerID="49b296095d99cebbcb1cf079e727995b87590d86300fbcf6e1c33012c4842652" Oct 31 19:50:58.816943 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:50:58.816922 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49b296095d99cebbcb1cf079e727995b87590d86300fbcf6e1c33012c4842652\": container with ID starting with 49b296095d99cebbcb1cf079e727995b87590d86300fbcf6e1c33012c4842652 not found: ID does not exist" containerID="49b296095d99cebbcb1cf079e727995b87590d86300fbcf6e1c33012c4842652" Oct 31 19:50:58.817032 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.816955 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49b296095d99cebbcb1cf079e727995b87590d86300fbcf6e1c33012c4842652"} err="failed to get container status \"49b296095d99cebbcb1cf079e727995b87590d86300fbcf6e1c33012c4842652\": rpc error: code = NotFound desc = could not find container \"49b296095d99cebbcb1cf079e727995b87590d86300fbcf6e1c33012c4842652\": container with ID starting with 49b296095d99cebbcb1cf079e727995b87590d86300fbcf6e1c33012c4842652 not found: ID does not exist" Oct 31 19:50:58.817032 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.816977 4910 scope.go:122] "RemoveContainer" containerID="6f3b83ca6f6ad605de24d7751c2606b03ebbe6624508c6c6865d37a6e9523cbb" Oct 31 19:50:58.817252 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:50:58.817231 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f3b83ca6f6ad605de24d7751c2606b03ebbe6624508c6c6865d37a6e9523cbb\": container with ID starting with 6f3b83ca6f6ad605de24d7751c2606b03ebbe6624508c6c6865d37a6e9523cbb not found: ID does not exist" containerID="6f3b83ca6f6ad605de24d7751c2606b03ebbe6624508c6c6865d37a6e9523cbb" Oct 31 19:50:58.817322 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.817261 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f3b83ca6f6ad605de24d7751c2606b03ebbe6624508c6c6865d37a6e9523cbb"} err="failed to get container status \"6f3b83ca6f6ad605de24d7751c2606b03ebbe6624508c6c6865d37a6e9523cbb\": rpc error: code = NotFound desc = could not find container \"6f3b83ca6f6ad605de24d7751c2606b03ebbe6624508c6c6865d37a6e9523cbb\": container with ID starting with 6f3b83ca6f6ad605de24d7751c2606b03ebbe6624508c6c6865d37a6e9523cbb not found: ID does not exist" Oct 31 19:50:58.831700 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.831660 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq"] Oct 31 19:50:58.836509 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:58.836477 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b1fae7-openshift-default-697b4b59f-jxhzq"] Oct 31 19:50:59.760368 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:59.760330 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f646f52-3c80-4fbb-8049-712f1e6b1595" path="/var/lib/kubelet/pods/9f646f52-3c80-4fbb-8049-712f1e6b1595/volumes" Oct 31 19:50:59.760954 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:50:59.760930 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be3bf6bf-4863-4489-b924-3de83b589c8e" path="/var/lib/kubelet/pods/be3bf6bf-4863-4489-b924-3de83b589c8e/volumes" Oct 31 19:51:01.224173 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.223026 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx"] Oct 31 19:51:01.224173 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.223354 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" podUID="73aa5edd-70dd-4bf1-b51c-ba31774baacd" containerName="istio-proxy" containerID="cri-o://4c4da181b1412575aa17cdee271cab49c42f6fffa7433b1400012c4ed753756c" gracePeriod=30 Oct 31 19:51:01.229889 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.229861 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h"] Oct 31 19:51:01.230089 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.230066 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" podUID="5e8d77e7-9866-404f-83fc-1809898cec90" containerName="istio-proxy" containerID="cri-o://241c1ff159899727ff021baf6d388074d37d7e94d525a6520f56d1bb22a8195b" gracePeriod=30 Oct 31 19:51:01.805152 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.805117 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db"] Oct 31 19:51:01.805962 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.805939 4910 cpu_manager.go:414] "containerMap: removing container" podUID="9f646f52-3c80-4fbb-8049-712f1e6b1595" containerName="istio-proxy" Oct 31 19:51:01.805962 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.805959 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f646f52-3c80-4fbb-8049-712f1e6b1595" containerName="istio-proxy" Oct 31 19:51:01.806129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.805973 4910 cpu_manager.go:414] "containerMap: removing container" podUID="706169ef-5dc5-4533-86af-fd02de456ab0" containerName="istio-proxy" Oct 31 19:51:01.806129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.805978 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="706169ef-5dc5-4533-86af-fd02de456ab0" containerName="istio-proxy" Oct 31 19:51:01.806129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.806024 4910 cpu_manager.go:414] "containerMap: removing container" podUID="be3bf6bf-4863-4489-b924-3de83b589c8e" containerName="istio-proxy" Oct 31 19:51:01.806129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.806029 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be3bf6bf-4863-4489-b924-3de83b589c8e" containerName="istio-proxy" Oct 31 19:51:01.806129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.806039 4910 cpu_manager.go:414] "containerMap: removing container" podUID="706169ef-5dc5-4533-86af-fd02de456ab0" containerName="istio-proxy" Oct 31 19:51:01.806129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.806044 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="706169ef-5dc5-4533-86af-fd02de456ab0" containerName="istio-proxy" Oct 31 19:51:01.806129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.806062 4910 cpu_manager.go:414] "containerMap: removing container" podUID="76ca271f-5d14-4807-928d-a4c5c23debc4" containerName="revoke-keys" Oct 31 19:51:01.806129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.806066 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ca271f-5d14-4807-928d-a4c5c23debc4" containerName="revoke-keys" Oct 31 19:51:01.806129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.806084 4910 cpu_manager.go:414] "containerMap: removing container" podUID="5928c0b9-c985-4b45-bd5e-b0543e262336" containerName="revoke-keys" Oct 31 19:51:01.806129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.806089 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5928c0b9-c985-4b45-bd5e-b0543e262336" containerName="revoke-keys" Oct 31 19:51:01.806436 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.806193 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="be3bf6bf-4863-4489-b924-3de83b589c8e" containerName="istio-proxy" Oct 31 19:51:01.806436 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.806203 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="706169ef-5dc5-4533-86af-fd02de456ab0" containerName="istio-proxy" Oct 31 19:51:01.806436 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.806212 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="499b41f2-15df-4636-abbc-35fa6f8a164a" containerName="istio-proxy" Oct 31 19:51:01.806436 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.806218 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ca271f-5d14-4807-928d-a4c5c23debc4" containerName="revoke-keys" Oct 31 19:51:01.806436 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.806226 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="5928c0b9-c985-4b45-bd5e-b0543e262336" containerName="revoke-keys" Oct 31 19:51:01.806436 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.806233 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f646f52-3c80-4fbb-8049-712f1e6b1595" containerName="istio-proxy" Oct 31 19:51:01.806436 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.806240 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="706169ef-5dc5-4533-86af-fd02de456ab0" containerName="istio-proxy" Oct 31 19:51:01.806436 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.806247 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f646f52-3c80-4fbb-8049-712f1e6b1595" containerName="istio-proxy" Oct 31 19:51:01.811582 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.811564 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db" Oct 31 19:51:01.829718 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.829608 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:51:01.829718 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.829684 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:51:01.836526 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.836494 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db"] Oct 31 19:51:01.878385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.878347 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fcfd21e-6616-4e59-a233-57f614e07231-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db\" (UID: \"5fcfd21e-6616-4e59-a233-57f614e07231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db" Oct 31 19:51:01.878590 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.878562 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcfk5\" (UniqueName: \"kubernetes.io/projected/5fcfd21e-6616-4e59-a233-57f614e07231-kube-api-access-xcfk5\") pod \"maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db\" (UID: \"5fcfd21e-6616-4e59-a233-57f614e07231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db" Oct 31 19:51:01.878717 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.878692 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fcfd21e-6616-4e59-a233-57f614e07231-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db\" (UID: \"5fcfd21e-6616-4e59-a233-57f614e07231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db" Oct 31 19:51:01.980717 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.980683 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcfk5\" (UniqueName: \"kubernetes.io/projected/5fcfd21e-6616-4e59-a233-57f614e07231-kube-api-access-xcfk5\") pod \"maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db\" (UID: \"5fcfd21e-6616-4e59-a233-57f614e07231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db" Oct 31 19:51:01.980961 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.980844 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fcfd21e-6616-4e59-a233-57f614e07231-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db\" (UID: \"5fcfd21e-6616-4e59-a233-57f614e07231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db" Oct 31 19:51:01.981159 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.981139 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fcfd21e-6616-4e59-a233-57f614e07231-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db\" (UID: \"5fcfd21e-6616-4e59-a233-57f614e07231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db" Oct 31 19:51:01.981481 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.981465 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fcfd21e-6616-4e59-a233-57f614e07231-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db\" (UID: \"5fcfd21e-6616-4e59-a233-57f614e07231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db" Oct 31 19:51:01.983444 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.983424 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fcfd21e-6616-4e59-a233-57f614e07231-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db\" (UID: \"5fcfd21e-6616-4e59-a233-57f614e07231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db" Oct 31 19:51:01.996649 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:01.996626 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcfk5\" (UniqueName: \"kubernetes.io/projected/5fcfd21e-6616-4e59-a233-57f614e07231-kube-api-access-xcfk5\") pod \"maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db\" (UID: \"5fcfd21e-6616-4e59-a233-57f614e07231\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db" Oct 31 19:51:02.130413 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:02.130380 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db" Oct 31 19:51:02.489692 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:02.489606 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db"] Oct 31 19:51:02.841988 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:02.841947 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db" event={"ID":"5fcfd21e-6616-4e59-a233-57f614e07231","Type":"ContainerStarted","Data":"e6cbf40715129591460ed04af32b111de1387a657bcf22d165ee3497112524fb"} Oct 31 19:51:03.040286 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:03.040253 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j" podUID="09ee3f2a-332e-4c98-9b21-c76f3903b69f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:51:03.117596 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:03.117518 4910 patch_prober.go:29] interesting pod/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:51:03.117596 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:03.117579 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" podUID="5e8d77e7-9866-404f-83fc-1809898cec90" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:51:03.960998 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:03.960382 4910 generic.go:356] "Generic (PLEG): container finished" podID="5fcfd21e-6616-4e59-a233-57f614e07231" containerID="02c8e866d7404627aea4c9ebe53ac88fb863f7478a988313b31894abf8afa24b" exitCode=0 Oct 31 19:51:03.960998 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:03.960680 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db" event={"ID":"5fcfd21e-6616-4e59-a233-57f614e07231","Type":"ContainerDied","Data":"02c8e866d7404627aea4c9ebe53ac88fb863f7478a988313b31894abf8afa24b"} Oct 31 19:51:04.881363 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:04.881325 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn"] Oct 31 19:51:04.883149 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:04.883123 4910 cpu_manager.go:414] "containerMap: removing container" podUID="9f646f52-3c80-4fbb-8049-712f1e6b1595" containerName="istio-proxy" Oct 31 19:51:04.883327 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:04.883313 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f646f52-3c80-4fbb-8049-712f1e6b1595" containerName="istio-proxy" Oct 31 19:51:04.889123 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:04.889097 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn" Oct 31 19:51:04.957638 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:04.957596 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn"] Oct 31 19:51:05.001222 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:05.001154 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d3ccc1-b910-4e07-8d4a-8a55e19070a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn\" (UID: \"c4d3ccc1-b910-4e07-8d4a-8a55e19070a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn" Oct 31 19:51:05.001648 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:05.001452 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd8jg\" (UniqueName: \"kubernetes.io/projected/c4d3ccc1-b910-4e07-8d4a-8a55e19070a5-kube-api-access-wd8jg\") pod \"maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn\" (UID: \"c4d3ccc1-b910-4e07-8d4a-8a55e19070a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn" Oct 31 19:51:05.001648 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:05.001517 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d3ccc1-b910-4e07-8d4a-8a55e19070a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn\" (UID: \"c4d3ccc1-b910-4e07-8d4a-8a55e19070a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn" Oct 31 19:51:05.103126 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:05.103085 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d3ccc1-b910-4e07-8d4a-8a55e19070a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn\" (UID: \"c4d3ccc1-b910-4e07-8d4a-8a55e19070a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn" Oct 31 19:51:05.103371 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:05.103337 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd8jg\" (UniqueName: \"kubernetes.io/projected/c4d3ccc1-b910-4e07-8d4a-8a55e19070a5-kube-api-access-wd8jg\") pod \"maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn\" (UID: \"c4d3ccc1-b910-4e07-8d4a-8a55e19070a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn" Oct 31 19:51:05.103472 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:05.103388 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d3ccc1-b910-4e07-8d4a-8a55e19070a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn\" (UID: \"c4d3ccc1-b910-4e07-8d4a-8a55e19070a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn" Oct 31 19:51:05.103994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:05.103906 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d3ccc1-b910-4e07-8d4a-8a55e19070a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn\" (UID: \"c4d3ccc1-b910-4e07-8d4a-8a55e19070a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn" Oct 31 19:51:05.105997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:05.105975 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d3ccc1-b910-4e07-8d4a-8a55e19070a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn\" (UID: \"c4d3ccc1-b910-4e07-8d4a-8a55e19070a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn" Oct 31 19:51:05.124559 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:05.124531 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd8jg\" (UniqueName: \"kubernetes.io/projected/c4d3ccc1-b910-4e07-8d4a-8a55e19070a5-kube-api-access-wd8jg\") pod \"maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn\" (UID: \"c4d3ccc1-b910-4e07-8d4a-8a55e19070a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn" Oct 31 19:51:05.143292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:05.143224 4910 patch_prober.go:29] interesting pod/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:51:05.143292 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:05.143278 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" podUID="73aa5edd-70dd-4bf1-b51c-ba31774baacd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:51:05.186674 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:05.186648 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db" Oct 31 19:51:05.278054 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:05.278018 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn" Oct 31 19:51:05.307068 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:05.307037 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fcfd21e-6616-4e59-a233-57f614e07231-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fcfd21e-6616-4e59-a233-57f614e07231-maas-api-openshift-ca\") pod \"5fcfd21e-6616-4e59-a233-57f614e07231\" (UID: \"5fcfd21e-6616-4e59-a233-57f614e07231\") " Oct 31 19:51:05.307197 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:05.307091 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fcfd21e-6616-4e59-a233-57f614e07231-kube-api-access-xcfk5\" (UniqueName: \"kubernetes.io/projected/5fcfd21e-6616-4e59-a233-57f614e07231-kube-api-access-xcfk5\") pod \"5fcfd21e-6616-4e59-a233-57f614e07231\" (UID: \"5fcfd21e-6616-4e59-a233-57f614e07231\") " Oct 31 19:51:05.307197 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:05.307138 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fcfd21e-6616-4e59-a233-57f614e07231-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fcfd21e-6616-4e59-a233-57f614e07231-maas-api-xks-ca\") pod \"5fcfd21e-6616-4e59-a233-57f614e07231\" (UID: \"5fcfd21e-6616-4e59-a233-57f614e07231\") " Oct 31 19:51:05.307478 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:05.307452 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fcfd21e-6616-4e59-a233-57f614e07231-maas-api-openshift-ca" pod "5fcfd21e-6616-4e59-a233-57f614e07231" (UID: "5fcfd21e-6616-4e59-a233-57f614e07231"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:51:05.308155 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:05.308120 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fcfd21e-6616-4e59-a233-57f614e07231-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:05.309796 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:05.309771 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fcfd21e-6616-4e59-a233-57f614e07231-maas-api-xks-ca" pod "5fcfd21e-6616-4e59-a233-57f614e07231" (UID: "5fcfd21e-6616-4e59-a233-57f614e07231"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:51:05.309891 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:05.309773 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fcfd21e-6616-4e59-a233-57f614e07231-kube-api-access-xcfk5" pod "5fcfd21e-6616-4e59-a233-57f614e07231" (UID: "5fcfd21e-6616-4e59-a233-57f614e07231"). InnerVolumeSpecName "kube-api-access-xcfk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:05.411556 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:05.411478 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcfk5\" (UniqueName: \"kubernetes.io/projected/5fcfd21e-6616-4e59-a233-57f614e07231-kube-api-access-xcfk5\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:05.411556 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:05.411520 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fcfd21e-6616-4e59-a233-57f614e07231-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:05.577742 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:51:05.577711 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d3ccc1_b910_4e07_8d4a_8a55e19070a5.slice/crio-934983cb293d3e5a14653e2d564315c561293404696e7b65d88797ee528e8cae WatchSource:0}: Error finding container 934983cb293d3e5a14653e2d564315c561293404696e7b65d88797ee528e8cae: Status 404 returned error can't find the container with id 934983cb293d3e5a14653e2d564315c561293404696e7b65d88797ee528e8cae Oct 31 19:51:05.579051 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:05.579021 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn"] Oct 31 19:51:06.056562 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.056473 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db" event={"ID":"5fcfd21e-6616-4e59-a233-57f614e07231","Type":"ContainerDied","Data":"e6cbf40715129591460ed04af32b111de1387a657bcf22d165ee3497112524fb"} Oct 31 19:51:06.057077 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.056564 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6cbf40715129591460ed04af32b111de1387a657bcf22d165ee3497112524fb" Oct 31 19:51:06.057077 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.056631 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db" Oct 31 19:51:06.059498 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.059462 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn" event={"ID":"c4d3ccc1-b910-4e07-8d4a-8a55e19070a5","Type":"ContainerStarted","Data":"934983cb293d3e5a14653e2d564315c561293404696e7b65d88797ee528e8cae"} Oct 31 19:51:06.405525 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.405486 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j" Oct 31 19:51:06.459530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.459487 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ee3f2a-332e-4c98-9b21-c76f3903b69f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09ee3f2a-332e-4c98-9b21-c76f3903b69f-plugins-config-volume\") pod \"09ee3f2a-332e-4c98-9b21-c76f3903b69f\" (UID: \"09ee3f2a-332e-4c98-9b21-c76f3903b69f\") " Oct 31 19:51:06.459830 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.459799 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ee3f2a-332e-4c98-9b21-c76f3903b69f-kube-api-access-9hb4g\" (UniqueName: \"kubernetes.io/projected/09ee3f2a-332e-4c98-9b21-c76f3903b69f-kube-api-access-9hb4g\") pod \"09ee3f2a-332e-4c98-9b21-c76f3903b69f\" (UID: \"09ee3f2a-332e-4c98-9b21-c76f3903b69f\") " Oct 31 19:51:06.460092 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.460066 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ee3f2a-332e-4c98-9b21-c76f3903b69f-plugins-config-volume" pod "09ee3f2a-332e-4c98-9b21-c76f3903b69f" (UID: "09ee3f2a-332e-4c98-9b21-c76f3903b69f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:51:06.460785 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.460756 4910 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09ee3f2a-332e-4c98-9b21-c76f3903b69f-plugins-config-volume\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:06.463422 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.463383 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ee3f2a-332e-4c98-9b21-c76f3903b69f-kube-api-access-9hb4g" pod "09ee3f2a-332e-4c98-9b21-c76f3903b69f" (UID: "09ee3f2a-332e-4c98-9b21-c76f3903b69f"). InnerVolumeSpecName "kube-api-access-9hb4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:06.562594 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.562554 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hb4g\" (UniqueName: \"kubernetes.io/projected/09ee3f2a-332e-4c98-9b21-c76f3903b69f-kube-api-access-9hb4g\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:06.583163 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.583094 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:51:06.607740 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.607715 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:51:06.665935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.665340 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73aa5edd-70dd-4bf1-b51c-ba31774baacd-kube-api-access-h668j\" (UniqueName: \"kubernetes.io/projected/73aa5edd-70dd-4bf1-b51c-ba31774baacd-kube-api-access-h668j\") pod \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " Oct 31 19:51:06.665935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.665406 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-workload-socket\") pod \"5e8d77e7-9866-404f-83fc-1809898cec90\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " Oct 31 19:51:06.665935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.665480 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e8d77e7-9866-404f-83fc-1809898cec90-kube-api-access-77wml\" (UniqueName: \"kubernetes.io/projected/5e8d77e7-9866-404f-83fc-1809898cec90-kube-api-access-77wml\") pod \"5e8d77e7-9866-404f-83fc-1809898cec90\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " Oct 31 19:51:06.665935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.665512 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-workload-socket\") pod \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " Oct 31 19:51:06.665935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.665577 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-workload-certs\") pod \"5e8d77e7-9866-404f-83fc-1809898cec90\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " Oct 31 19:51:06.665935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.665627 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istiod-ca-cert\") pod \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " Oct 31 19:51:06.665935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.665685 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e8d77e7-9866-404f-83fc-1809898cec90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8d77e7-9866-404f-83fc-1809898cec90-istio-podinfo\") pod \"5e8d77e7-9866-404f-83fc-1809898cec90\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " Oct 31 19:51:06.665935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.665720 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-data\") pod \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " Oct 31 19:51:06.665935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.665757 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-credential-socket\") pod \"5e8d77e7-9866-404f-83fc-1809898cec90\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " Oct 31 19:51:06.665935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.665809 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-istio-data\") pod \"5e8d77e7-9866-404f-83fc-1809898cec90\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " Oct 31 19:51:06.665935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.665845 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e8d77e7-9866-404f-83fc-1809898cec90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8d77e7-9866-404f-83fc-1809898cec90-istiod-ca-cert\") pod \"5e8d77e7-9866-404f-83fc-1809898cec90\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " Oct 31 19:51:06.665935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.665943 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-istio-envoy\") pod \"5e8d77e7-9866-404f-83fc-1809898cec90\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " Oct 31 19:51:06.666686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.665997 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-token\" (UniqueName: \"kubernetes.io/projected/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-token\") pod \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " Oct 31 19:51:06.666686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.666017 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-workload-socket" pod "73aa5edd-70dd-4bf1-b51c-ba31774baacd" (UID: "73aa5edd-70dd-4bf1-b51c-ba31774baacd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:06.666686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.666049 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-envoy\") pod \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " Oct 31 19:51:06.666686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.666088 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-credential-socket\") pod \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " Oct 31 19:51:06.666686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.666122 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-podinfo\") pod \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " Oct 31 19:51:06.666686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.666321 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e8d77e7-9866-404f-83fc-1809898cec90-istio-token\" (UniqueName: \"kubernetes.io/projected/5e8d77e7-9866-404f-83fc-1809898cec90-istio-token\") pod \"5e8d77e7-9866-404f-83fc-1809898cec90\" (UID: \"5e8d77e7-9866-404f-83fc-1809898cec90\") " Oct 31 19:51:06.666686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.666382 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-workload-certs\") pod \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\" (UID: \"73aa5edd-70dd-4bf1-b51c-ba31774baacd\") " Oct 31 19:51:06.667070 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.666901 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-workload-socket" pod "5e8d77e7-9866-404f-83fc-1809898cec90" (UID: "5e8d77e7-9866-404f-83fc-1809898cec90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:06.667129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.667103 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-workload-certs" pod "73aa5edd-70dd-4bf1-b51c-ba31774baacd" (UID: "73aa5edd-70dd-4bf1-b51c-ba31774baacd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:06.667475 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.667429 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:06.667475 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.667461 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:06.667650 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.667478 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:06.667709 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.667664 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-credential-socket" pod "73aa5edd-70dd-4bf1-b51c-ba31774baacd" (UID: "73aa5edd-70dd-4bf1-b51c-ba31774baacd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:06.668151 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.667754 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e8d77e7-9866-404f-83fc-1809898cec90-istiod-ca-cert" pod "5e8d77e7-9866-404f-83fc-1809898cec90" (UID: "5e8d77e7-9866-404f-83fc-1809898cec90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:51:06.668151 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.668051 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-workload-certs" pod "5e8d77e7-9866-404f-83fc-1809898cec90" (UID: "5e8d77e7-9866-404f-83fc-1809898cec90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:06.668375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.668354 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istiod-ca-cert" pod "73aa5edd-70dd-4bf1-b51c-ba31774baacd" (UID: "73aa5edd-70dd-4bf1-b51c-ba31774baacd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:51:06.668832 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.668808 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-credential-socket" pod "5e8d77e7-9866-404f-83fc-1809898cec90" (UID: "5e8d77e7-9866-404f-83fc-1809898cec90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:06.669110 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.669090 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-data" pod "73aa5edd-70dd-4bf1-b51c-ba31774baacd" (UID: "73aa5edd-70dd-4bf1-b51c-ba31774baacd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:06.669396 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.669373 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-istio-data" pod "5e8d77e7-9866-404f-83fc-1809898cec90" (UID: "5e8d77e7-9866-404f-83fc-1809898cec90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:06.670739 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.670717 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e8d77e7-9866-404f-83fc-1809898cec90-istio-podinfo" pod "5e8d77e7-9866-404f-83fc-1809898cec90" (UID: "5e8d77e7-9866-404f-83fc-1809898cec90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:51:06.672359 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.672226 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-envoy" pod "73aa5edd-70dd-4bf1-b51c-ba31774baacd" (UID: "73aa5edd-70dd-4bf1-b51c-ba31774baacd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:06.672359 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.672341 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8d77e7-9866-404f-83fc-1809898cec90-istio-token" pod "5e8d77e7-9866-404f-83fc-1809898cec90" (UID: "5e8d77e7-9866-404f-83fc-1809898cec90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:06.672557 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.672535 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8d77e7-9866-404f-83fc-1809898cec90-kube-api-access-77wml" pod "5e8d77e7-9866-404f-83fc-1809898cec90" (UID: "5e8d77e7-9866-404f-83fc-1809898cec90"). InnerVolumeSpecName "kube-api-access-77wml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:06.673578 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.673557 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-podinfo" pod "73aa5edd-70dd-4bf1-b51c-ba31774baacd" (UID: "73aa5edd-70dd-4bf1-b51c-ba31774baacd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:51:06.677274 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.677189 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-istio-envoy" pod "5e8d77e7-9866-404f-83fc-1809898cec90" (UID: "5e8d77e7-9866-404f-83fc-1809898cec90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:06.677274 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.677229 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73aa5edd-70dd-4bf1-b51c-ba31774baacd-kube-api-access-h668j" pod "73aa5edd-70dd-4bf1-b51c-ba31774baacd" (UID: "73aa5edd-70dd-4bf1-b51c-ba31774baacd"). InnerVolumeSpecName "kube-api-access-h668j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:06.685806 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.685765 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-token" pod "73aa5edd-70dd-4bf1-b51c-ba31774baacd" (UID: "73aa5edd-70dd-4bf1-b51c-ba31774baacd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:06.769715 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.769678 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e8d77e7-9866-404f-83fc-1809898cec90-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:06.769897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.769722 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h668j\" (UniqueName: \"kubernetes.io/projected/73aa5edd-70dd-4bf1-b51c-ba31774baacd-kube-api-access-h668j\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:06.769897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.769741 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77wml\" (UniqueName: \"kubernetes.io/projected/5e8d77e7-9866-404f-83fc-1809898cec90-kube-api-access-77wml\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:06.769897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.769756 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:06.769897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.769770 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:06.769897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.769784 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e8d77e7-9866-404f-83fc-1809898cec90-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:06.769897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.769799 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:06.769897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.769813 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:06.769897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.769827 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:06.769897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.769842 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e8d77e7-9866-404f-83fc-1809898cec90-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:06.769897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.769857 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e8d77e7-9866-404f-83fc-1809898cec90-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:06.769897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.769870 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:06.769897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.769883 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:06.769897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.769899 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73aa5edd-70dd-4bf1-b51c-ba31774baacd-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:06.770553 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:06.770011 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73aa5edd-70dd-4bf1-b51c-ba31774baacd-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:07.071992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.071948 4910 generic.go:356] "Generic (PLEG): container finished" podID="73aa5edd-70dd-4bf1-b51c-ba31774baacd" containerID="4c4da181b1412575aa17cdee271cab49c42f6fffa7433b1400012c4ed753756c" exitCode=0 Oct 31 19:51:07.072466 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.072002 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" Oct 31 19:51:07.072466 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.072002 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" event={"ID":"73aa5edd-70dd-4bf1-b51c-ba31774baacd","Type":"ContainerDied","Data":"4c4da181b1412575aa17cdee271cab49c42f6fffa7433b1400012c4ed753756c"} Oct 31 19:51:07.072466 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.072057 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx" event={"ID":"73aa5edd-70dd-4bf1-b51c-ba31774baacd","Type":"ContainerDied","Data":"829b63d1304303165e998b19b6f0533b2844e98a282b813938c9b4f0eb472463"} Oct 31 19:51:07.072466 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.072084 4910 scope.go:122] "RemoveContainer" containerID="4c4da181b1412575aa17cdee271cab49c42f6fffa7433b1400012c4ed753756c" Oct 31 19:51:07.073732 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.073701 4910 generic.go:356] "Generic (PLEG): container finished" podID="09ee3f2a-332e-4c98-9b21-c76f3903b69f" containerID="cd2371917831ea70df2741d58567ef8a93339d01d1b41ae865a4415ed9c429d3" exitCode=0 Oct 31 19:51:07.073872 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.073753 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j" Oct 31 19:51:07.073872 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.073782 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j" event={"ID":"09ee3f2a-332e-4c98-9b21-c76f3903b69f","Type":"ContainerDied","Data":"cd2371917831ea70df2741d58567ef8a93339d01d1b41ae865a4415ed9c429d3"} Oct 31 19:51:07.073872 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.073845 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j" event={"ID":"09ee3f2a-332e-4c98-9b21-c76f3903b69f","Type":"ContainerDied","Data":"7eef78e5d7e075acd5c2514123103b7c0008409819e59ab6886362a27ff34cdc"} Oct 31 19:51:07.075950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.075881 4910 generic.go:356] "Generic (PLEG): container finished" podID="c4d3ccc1-b910-4e07-8d4a-8a55e19070a5" containerID="3a9ec0241f40c48629c94ee14d8fbe3fcfda70d4efb6e9ca798fa4c7acb3b34e" exitCode=0 Oct 31 19:51:07.076073 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.076031 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn" event={"ID":"c4d3ccc1-b910-4e07-8d4a-8a55e19070a5","Type":"ContainerDied","Data":"3a9ec0241f40c48629c94ee14d8fbe3fcfda70d4efb6e9ca798fa4c7acb3b34e"} Oct 31 19:51:07.078604 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.078575 4910 generic.go:356] "Generic (PLEG): container finished" podID="5e8d77e7-9866-404f-83fc-1809898cec90" containerID="241c1ff159899727ff021baf6d388074d37d7e94d525a6520f56d1bb22a8195b" exitCode=0 Oct 31 19:51:07.078697 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.078630 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" event={"ID":"5e8d77e7-9866-404f-83fc-1809898cec90","Type":"ContainerDied","Data":"241c1ff159899727ff021baf6d388074d37d7e94d525a6520f56d1bb22a8195b"} Oct 31 19:51:07.078697 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.078680 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" event={"ID":"5e8d77e7-9866-404f-83fc-1809898cec90","Type":"ContainerDied","Data":"ac5511460855bd3e01ada49cb531b58760d955e85174d46329d967e0a90c0591"} Oct 31 19:51:07.078814 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.078785 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h" Oct 31 19:51:07.130807 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.130774 4910 scope.go:122] "RemoveContainer" containerID="4c4da181b1412575aa17cdee271cab49c42f6fffa7433b1400012c4ed753756c" Oct 31 19:51:07.132027 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:51:07.131998 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c4da181b1412575aa17cdee271cab49c42f6fffa7433b1400012c4ed753756c\": container with ID starting with 4c4da181b1412575aa17cdee271cab49c42f6fffa7433b1400012c4ed753756c not found: ID does not exist" containerID="4c4da181b1412575aa17cdee271cab49c42f6fffa7433b1400012c4ed753756c" Oct 31 19:51:07.132260 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.132223 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c4da181b1412575aa17cdee271cab49c42f6fffa7433b1400012c4ed753756c"} err="failed to get container status \"4c4da181b1412575aa17cdee271cab49c42f6fffa7433b1400012c4ed753756c\": rpc error: code = NotFound desc = could not find container \"4c4da181b1412575aa17cdee271cab49c42f6fffa7433b1400012c4ed753756c\": container with ID starting with 4c4da181b1412575aa17cdee271cab49c42f6fffa7433b1400012c4ed753756c not found: ID does not exist" Oct 31 19:51:07.132401 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.132384 4910 scope.go:122] "RemoveContainer" containerID="cd2371917831ea70df2741d58567ef8a93339d01d1b41ae865a4415ed9c429d3" Oct 31 19:51:07.174206 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.174064 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx"] Oct 31 19:51:07.184334 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.184305 4910 scope.go:122] "RemoveContainer" containerID="cd2371917831ea70df2741d58567ef8a93339d01d1b41ae865a4415ed9c429d3" Oct 31 19:51:07.184959 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:51:07.184809 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd2371917831ea70df2741d58567ef8a93339d01d1b41ae865a4415ed9c429d3\": container with ID starting with cd2371917831ea70df2741d58567ef8a93339d01d1b41ae865a4415ed9c429d3 not found: ID does not exist" containerID="cd2371917831ea70df2741d58567ef8a93339d01d1b41ae865a4415ed9c429d3" Oct 31 19:51:07.184959 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.184851 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd2371917831ea70df2741d58567ef8a93339d01d1b41ae865a4415ed9c429d3"} err="failed to get container status \"cd2371917831ea70df2741d58567ef8a93339d01d1b41ae865a4415ed9c429d3\": rpc error: code = NotFound desc = could not find container \"cd2371917831ea70df2741d58567ef8a93339d01d1b41ae865a4415ed9c429d3\": container with ID starting with cd2371917831ea70df2741d58567ef8a93339d01d1b41ae865a4415ed9c429d3 not found: ID does not exist" Oct 31 19:51:07.184959 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.184880 4910 scope.go:122] "RemoveContainer" containerID="241c1ff159899727ff021baf6d388074d37d7e94d525a6520f56d1bb22a8195b" Oct 31 19:51:07.193449 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.193427 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-t2vcx"] Oct 31 19:51:07.198487 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.198464 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j"] Oct 31 19:51:07.201525 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.201482 4910 scope.go:122] "RemoveContainer" containerID="887e5ff8622f885dc948a0d730a1d9c5e85176dbf723a89aaf4433bfd83853b6" Oct 31 19:51:07.202566 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.202521 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a4e312-74585c8467-r5q8j"] Oct 31 19:51:07.212449 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.212433 4910 scope.go:122] "RemoveContainer" containerID="241c1ff159899727ff021baf6d388074d37d7e94d525a6520f56d1bb22a8195b" Oct 31 19:51:07.212743 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:51:07.212722 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"241c1ff159899727ff021baf6d388074d37d7e94d525a6520f56d1bb22a8195b\": container with ID starting with 241c1ff159899727ff021baf6d388074d37d7e94d525a6520f56d1bb22a8195b not found: ID does not exist" containerID="241c1ff159899727ff021baf6d388074d37d7e94d525a6520f56d1bb22a8195b" Oct 31 19:51:07.212803 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.212750 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"241c1ff159899727ff021baf6d388074d37d7e94d525a6520f56d1bb22a8195b"} err="failed to get container status \"241c1ff159899727ff021baf6d388074d37d7e94d525a6520f56d1bb22a8195b\": rpc error: code = NotFound desc = could not find container \"241c1ff159899727ff021baf6d388074d37d7e94d525a6520f56d1bb22a8195b\": container with ID starting with 241c1ff159899727ff021baf6d388074d37d7e94d525a6520f56d1bb22a8195b not found: ID does not exist" Oct 31 19:51:07.212803 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.212767 4910 scope.go:122] "RemoveContainer" containerID="887e5ff8622f885dc948a0d730a1d9c5e85176dbf723a89aaf4433bfd83853b6" Oct 31 19:51:07.213071 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:51:07.213052 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"887e5ff8622f885dc948a0d730a1d9c5e85176dbf723a89aaf4433bfd83853b6\": container with ID starting with 887e5ff8622f885dc948a0d730a1d9c5e85176dbf723a89aaf4433bfd83853b6 not found: ID does not exist" containerID="887e5ff8622f885dc948a0d730a1d9c5e85176dbf723a89aaf4433bfd83853b6" Oct 31 19:51:07.213127 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.213080 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"887e5ff8622f885dc948a0d730a1d9c5e85176dbf723a89aaf4433bfd83853b6"} err="failed to get container status \"887e5ff8622f885dc948a0d730a1d9c5e85176dbf723a89aaf4433bfd83853b6\": rpc error: code = NotFound desc = could not find container \"887e5ff8622f885dc948a0d730a1d9c5e85176dbf723a89aaf4433bfd83853b6\": container with ID starting with 887e5ff8622f885dc948a0d730a1d9c5e85176dbf723a89aaf4433bfd83853b6 not found: ID does not exist" Oct 31 19:51:07.221770 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.221749 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h"] Oct 31 19:51:07.231213 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.231193 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a4e312-openshift-default-65c69cb84c-w9f5h"] Oct 31 19:51:07.760166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.760129 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09ee3f2a-332e-4c98-9b21-c76f3903b69f" path="/var/lib/kubelet/pods/09ee3f2a-332e-4c98-9b21-c76f3903b69f/volumes" Oct 31 19:51:07.760923 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.760887 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e8d77e7-9866-404f-83fc-1809898cec90" path="/var/lib/kubelet/pods/5e8d77e7-9866-404f-83fc-1809898cec90/volumes" Oct 31 19:51:07.761591 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:07.761571 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73aa5edd-70dd-4bf1-b51c-ba31774baacd" path="/var/lib/kubelet/pods/73aa5edd-70dd-4bf1-b51c-ba31774baacd/volumes" Oct 31 19:51:08.290024 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:08.289998 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn" Oct 31 19:51:08.398935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:08.398835 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4d3ccc1-b910-4e07-8d4a-8a55e19070a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d3ccc1-b910-4e07-8d4a-8a55e19070a5-maas-api-xks-ca\") pod \"c4d3ccc1-b910-4e07-8d4a-8a55e19070a5\" (UID: \"c4d3ccc1-b910-4e07-8d4a-8a55e19070a5\") " Oct 31 19:51:08.399170 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:08.399139 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4d3ccc1-b910-4e07-8d4a-8a55e19070a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d3ccc1-b910-4e07-8d4a-8a55e19070a5-maas-api-openshift-ca\") pod \"c4d3ccc1-b910-4e07-8d4a-8a55e19070a5\" (UID: \"c4d3ccc1-b910-4e07-8d4a-8a55e19070a5\") " Oct 31 19:51:08.399346 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:08.399328 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4d3ccc1-b910-4e07-8d4a-8a55e19070a5-kube-api-access-wd8jg\" (UniqueName: \"kubernetes.io/projected/c4d3ccc1-b910-4e07-8d4a-8a55e19070a5-kube-api-access-wd8jg\") pod \"c4d3ccc1-b910-4e07-8d4a-8a55e19070a5\" (UID: \"c4d3ccc1-b910-4e07-8d4a-8a55e19070a5\") " Oct 31 19:51:08.399500 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:08.399481 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d3ccc1-b910-4e07-8d4a-8a55e19070a5-maas-api-openshift-ca" pod "c4d3ccc1-b910-4e07-8d4a-8a55e19070a5" (UID: "c4d3ccc1-b910-4e07-8d4a-8a55e19070a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:51:08.399887 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:08.399870 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d3ccc1-b910-4e07-8d4a-8a55e19070a5-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:08.401030 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:08.401008 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4d3ccc1-b910-4e07-8d4a-8a55e19070a5-maas-api-xks-ca" pod "c4d3ccc1-b910-4e07-8d4a-8a55e19070a5" (UID: "c4d3ccc1-b910-4e07-8d4a-8a55e19070a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:51:08.401405 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:08.401385 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d3ccc1-b910-4e07-8d4a-8a55e19070a5-kube-api-access-wd8jg" pod "c4d3ccc1-b910-4e07-8d4a-8a55e19070a5" (UID: "c4d3ccc1-b910-4e07-8d4a-8a55e19070a5"). InnerVolumeSpecName "kube-api-access-wd8jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:08.501485 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:08.501454 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd8jg\" (UniqueName: \"kubernetes.io/projected/c4d3ccc1-b910-4e07-8d4a-8a55e19070a5-kube-api-access-wd8jg\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:08.501485 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:08.501480 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d3ccc1-b910-4e07-8d4a-8a55e19070a5-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:09.108343 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:09.108301 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn" event={"ID":"c4d3ccc1-b910-4e07-8d4a-8a55e19070a5","Type":"ContainerDied","Data":"934983cb293d3e5a14653e2d564315c561293404696e7b65d88797ee528e8cae"} Oct 31 19:51:09.108343 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:09.108331 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn" Oct 31 19:51:09.108604 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:09.108359 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="934983cb293d3e5a14653e2d564315c561293404696e7b65d88797ee528e8cae" Oct 31 19:51:10.261867 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:10.261750 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql"] Oct 31 19:51:10.267347 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:10.267314 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9efeae-cn4ql"] Oct 31 19:51:11.759011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:11.758973 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b97ba48f-5c91-4007-bb09-a54f35264beb" path="/var/lib/kubelet/pods/b97ba48f-5c91-4007-bb09-a54f35264beb/volumes" Oct 31 19:51:12.715981 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:51:12.715942 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:51:15.068459 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:15.068419 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb"] Oct 31 19:51:17.576856 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:17.576815 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8"] Oct 31 19:51:17.577377 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:17.577116 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8" podUID="d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f" containerName="maas-api" containerID="cri-o://fa781564f45b28cf918599d797185df7ebb2ec36fe6d067106b56d054ba19e07" gracePeriod=30 Oct 31 19:51:17.650076 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:17.641594 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8" podUID="d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.184:8443/health\": dial tcp 10.132.0.184:8443: connect: connection refused" Oct 31 19:51:18.042000 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.038954 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn"] Oct 31 19:51:18.049642 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.049520 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1e8cf0-jq6nn"] Oct 31 19:51:18.401065 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.401030 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8" Oct 31 19:51:18.448589 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.448521 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-kube-api-access-bssdn\" (UniqueName: \"kubernetes.io/projected/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-kube-api-access-bssdn\") pod \"d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f\" (UID: \"d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f\") " Oct 31 19:51:18.449040 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.449004 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-openshift-service-ca\") pod \"d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f\" (UID: \"d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f\") " Oct 31 19:51:18.449270 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.449243 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-metrics-certs\") pod \"d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f\" (UID: \"d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f\") " Oct 31 19:51:18.449417 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.449401 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-maas-api-tls\") pod \"d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f\" (UID: \"d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f\") " Oct 31 19:51:18.491252 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.484643 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-openshift-service-ca" pod "d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f" (UID: "d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:51:18.529494 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.529457 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-maas-api-tls" pod "d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f" (UID: "d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:51:18.552648 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.552589 4910 generic.go:356] "Generic (PLEG): container finished" podID="d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f" containerID="fa781564f45b28cf918599d797185df7ebb2ec36fe6d067106b56d054ba19e07" exitCode=0 Oct 31 19:51:18.552856 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.552847 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8" Oct 31 19:51:18.553511 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.553472 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft"] Oct 31 19:51:18.553664 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.553517 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8" event={"ID":"d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f","Type":"ContainerDied","Data":"fa781564f45b28cf918599d797185df7ebb2ec36fe6d067106b56d054ba19e07"} Oct 31 19:51:18.553664 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.553561 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8" event={"ID":"d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f","Type":"ContainerDied","Data":"3abb0e50e1a06e0372eb8f7af40ef6fc2d459385aeda13617584000a93913a5b"} Oct 31 19:51:18.553664 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.553592 4910 scope.go:122] "RemoveContainer" containerID="fa781564f45b28cf918599d797185df7ebb2ec36fe6d067106b56d054ba19e07" Oct 31 19:51:18.556262 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.556227 4910 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-openshift-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:18.559243 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.559217 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-maas-api-tls\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:18.566315 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.566257 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f8c3f7-f6mft"] Oct 31 19:51:18.568280 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.568203 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-kube-api-access-bssdn" pod "d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f" (UID: "d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f"). InnerVolumeSpecName "kube-api-access-bssdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:18.578331 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.578300 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-metrics-certs" pod "d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f" (UID: "d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:51:18.671991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.663932 4910 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-metrics-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:18.671991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.663981 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bssdn\" (UniqueName: \"kubernetes.io/projected/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f-kube-api-access-bssdn\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:18.671991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.664471 4910 scope.go:122] "RemoveContainer" containerID="fa781564f45b28cf918599d797185df7ebb2ec36fe6d067106b56d054ba19e07" Oct 31 19:51:18.703449 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:51:18.687056 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa781564f45b28cf918599d797185df7ebb2ec36fe6d067106b56d054ba19e07\": container with ID starting with fa781564f45b28cf918599d797185df7ebb2ec36fe6d067106b56d054ba19e07 not found: ID does not exist" containerID="fa781564f45b28cf918599d797185df7ebb2ec36fe6d067106b56d054ba19e07" Oct 31 19:51:18.703449 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.687277 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa781564f45b28cf918599d797185df7ebb2ec36fe6d067106b56d054ba19e07"} err="failed to get container status \"fa781564f45b28cf918599d797185df7ebb2ec36fe6d067106b56d054ba19e07\": rpc error: code = NotFound desc = could not find container \"fa781564f45b28cf918599d797185df7ebb2ec36fe6d067106b56d054ba19e07\": container with ID starting with fa781564f45b28cf918599d797185df7ebb2ec36fe6d067106b56d054ba19e07 not found: ID does not exist" Oct 31 19:51:18.892874 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.892629 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8"] Oct 31 19:51:18.898714 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:18.898685 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1e8cf0-7bfbf65c4f-6r2l8"] Oct 31 19:51:19.760660 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:19.760623 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5928c0b9-c985-4b45-bd5e-b0543e262336" path="/var/lib/kubelet/pods/5928c0b9-c985-4b45-bd5e-b0543e262336/volumes" Oct 31 19:51:19.761353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:19.761336 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4d3ccc1-b910-4e07-8d4a-8a55e19070a5" path="/var/lib/kubelet/pods/c4d3ccc1-b910-4e07-8d4a-8a55e19070a5/volumes" Oct 31 19:51:19.761905 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:19.761880 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f" path="/var/lib/kubelet/pods/d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f/volumes" Oct 31 19:51:20.068830 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:20.068785 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb" podUID="62d760d9-672c-4690-b5de-e8c6456c5431" containerName="payload-pre-processing" containerID="cri-o://1f8427e1acc867c2d79b638d77272c277c74ca79b003de08f87083d6aff05c9e" gracePeriod=25 Oct 31 19:51:23.823570 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:23.822996 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb" podUID="62d760d9-672c-4690-b5de-e8c6456c5431" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:51:26.370291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:26.370245 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b"] Oct 31 19:51:26.370842 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:26.370529 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b" podUID="69d2da79-5a06-4c5a-9185-58c9136a05cf" containerName="maas-api" containerID="cri-o://c1f10e919922d3effa2bbec320f65da467f106ffe1d2bf7a35e8a7154295d3cb" gracePeriod=30 Oct 31 19:51:26.935006 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:26.934970 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db"] Oct 31 19:51:27.036451 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:27.036405 4910 generic.go:356] "Generic (PLEG): container finished" podID="69d2da79-5a06-4c5a-9185-58c9136a05cf" containerID="c1f10e919922d3effa2bbec320f65da467f106ffe1d2bf7a35e8a7154295d3cb" exitCode=0 Oct 31 19:51:27.037083 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:27.037030 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b" event={"ID":"69d2da79-5a06-4c5a-9185-58c9136a05cf","Type":"ContainerDied","Data":"c1f10e919922d3effa2bbec320f65da467f106ffe1d2bf7a35e8a7154295d3cb"} Oct 31 19:51:27.105079 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:27.091391 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d45f6c-db7db"] Oct 31 19:51:27.329092 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:27.329060 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b" Oct 31 19:51:27.423437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:27.423385 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69d2da79-5a06-4c5a-9185-58c9136a05cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/69d2da79-5a06-4c5a-9185-58c9136a05cf-metrics-certs\") pod \"69d2da79-5a06-4c5a-9185-58c9136a05cf\" (UID: \"69d2da79-5a06-4c5a-9185-58c9136a05cf\") " Oct 31 19:51:27.423994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:27.423467 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69d2da79-5a06-4c5a-9185-58c9136a05cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d2da79-5a06-4c5a-9185-58c9136a05cf-openshift-service-ca\") pod \"69d2da79-5a06-4c5a-9185-58c9136a05cf\" (UID: \"69d2da79-5a06-4c5a-9185-58c9136a05cf\") " Oct 31 19:51:27.423994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:27.423655 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69d2da79-5a06-4c5a-9185-58c9136a05cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69d2da79-5a06-4c5a-9185-58c9136a05cf-maas-api-tls\") pod \"69d2da79-5a06-4c5a-9185-58c9136a05cf\" (UID: \"69d2da79-5a06-4c5a-9185-58c9136a05cf\") " Oct 31 19:51:27.423994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:27.423835 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69d2da79-5a06-4c5a-9185-58c9136a05cf-kube-api-access-jpvvf\" (UniqueName: \"kubernetes.io/projected/69d2da79-5a06-4c5a-9185-58c9136a05cf-kube-api-access-jpvvf\") pod \"69d2da79-5a06-4c5a-9185-58c9136a05cf\" (UID: \"69d2da79-5a06-4c5a-9185-58c9136a05cf\") " Oct 31 19:51:27.423994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:27.423864 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d2da79-5a06-4c5a-9185-58c9136a05cf-openshift-service-ca" pod "69d2da79-5a06-4c5a-9185-58c9136a05cf" (UID: "69d2da79-5a06-4c5a-9185-58c9136a05cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:51:27.424929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:27.424887 4910 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69d2da79-5a06-4c5a-9185-58c9136a05cf-openshift-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:27.425704 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:27.425681 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69d2da79-5a06-4c5a-9185-58c9136a05cf-maas-api-tls" pod "69d2da79-5a06-4c5a-9185-58c9136a05cf" (UID: "69d2da79-5a06-4c5a-9185-58c9136a05cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:51:27.425826 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:27.425803 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69d2da79-5a06-4c5a-9185-58c9136a05cf-metrics-certs" pod "69d2da79-5a06-4c5a-9185-58c9136a05cf" (UID: "69d2da79-5a06-4c5a-9185-58c9136a05cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:51:27.426890 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:27.426874 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d2da79-5a06-4c5a-9185-58c9136a05cf-kube-api-access-jpvvf" pod "69d2da79-5a06-4c5a-9185-58c9136a05cf" (UID: "69d2da79-5a06-4c5a-9185-58c9136a05cf"). InnerVolumeSpecName "kube-api-access-jpvvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:27.533940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:27.528011 4910 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69d2da79-5a06-4c5a-9185-58c9136a05cf-metrics-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:27.533940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:27.528050 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69d2da79-5a06-4c5a-9185-58c9136a05cf-maas-api-tls\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:27.533940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:27.528069 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpvvf\" (UniqueName: \"kubernetes.io/projected/69d2da79-5a06-4c5a-9185-58c9136a05cf-kube-api-access-jpvvf\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:27.758722 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:27.758689 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fcfd21e-6616-4e59-a233-57f614e07231" path="/var/lib/kubelet/pods/5fcfd21e-6616-4e59-a233-57f614e07231/volumes" Oct 31 19:51:28.142936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:28.142489 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b" event={"ID":"69d2da79-5a06-4c5a-9185-58c9136a05cf","Type":"ContainerDied","Data":"8484f97bb9fc9201ffaae0d40a1c59edff9947e84de18d881b2a80a786528f6d"} Oct 31 19:51:28.142936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:28.142578 4910 scope.go:122] "RemoveContainer" containerID="c1f10e919922d3effa2bbec320f65da467f106ffe1d2bf7a35e8a7154295d3cb" Oct 31 19:51:28.142936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:28.142791 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b" Oct 31 19:51:28.282935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:28.275049 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b"] Oct 31 19:51:28.374129 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:28.371580 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d45f6c-6fcfcfd759-mfv9b"] Oct 31 19:51:28.482834 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:28.481398 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb"] Oct 31 19:51:28.482834 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:28.482326 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" podUID="c86e67dd-5de9-4213-b26d-fcbc3bd90353" containerName="istio-proxy" containerID="cri-o://418d0eab141ae1e42b12d7018a9cbf0f5f2f5afaa4c005214239ea3eedec345e" gracePeriod=30 Oct 31 19:51:29.232146 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.232013 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5"] Oct 31 19:51:29.233311 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.233283 4910 cpu_manager.go:414] "containerMap: removing container" podUID="73aa5edd-70dd-4bf1-b51c-ba31774baacd" containerName="istio-proxy" Oct 31 19:51:29.233489 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.233476 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73aa5edd-70dd-4bf1-b51c-ba31774baacd" containerName="istio-proxy" Oct 31 19:51:29.233567 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.233559 4910 cpu_manager.go:414] "containerMap: removing container" podUID="5fcfd21e-6616-4e59-a233-57f614e07231" containerName="revoke-keys" Oct 31 19:51:29.233632 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.233624 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fcfd21e-6616-4e59-a233-57f614e07231" containerName="revoke-keys" Oct 31 19:51:29.233719 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.233711 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c4d3ccc1-b910-4e07-8d4a-8a55e19070a5" containerName="revoke-keys" Oct 31 19:51:29.233778 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.233770 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4d3ccc1-b910-4e07-8d4a-8a55e19070a5" containerName="revoke-keys" Oct 31 19:51:29.233858 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.233850 4910 cpu_manager.go:414] "containerMap: removing container" podUID="09ee3f2a-332e-4c98-9b21-c76f3903b69f" containerName="payload-pre-processing" Oct 31 19:51:29.233947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.233937 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ee3f2a-332e-4c98-9b21-c76f3903b69f" containerName="payload-pre-processing" Oct 31 19:51:29.234025 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.234017 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f" containerName="maas-api" Oct 31 19:51:29.234089 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.234081 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f" containerName="maas-api" Oct 31 19:51:29.234198 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.234188 4910 cpu_manager.go:414] "containerMap: removing container" podUID="5e8d77e7-9866-404f-83fc-1809898cec90" containerName="istio-proxy" Oct 31 19:51:29.234260 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.234251 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e8d77e7-9866-404f-83fc-1809898cec90" containerName="istio-proxy" Oct 31 19:51:29.234330 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.234322 4910 cpu_manager.go:414] "containerMap: removing container" podUID="5e8d77e7-9866-404f-83fc-1809898cec90" containerName="istio-proxy" Oct 31 19:51:29.234449 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.234434 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e8d77e7-9866-404f-83fc-1809898cec90" containerName="istio-proxy" Oct 31 19:51:29.234556 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.234547 4910 cpu_manager.go:414] "containerMap: removing container" podUID="69d2da79-5a06-4c5a-9185-58c9136a05cf" containerName="maas-api" Oct 31 19:51:29.234680 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.234667 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69d2da79-5a06-4c5a-9185-58c9136a05cf" containerName="maas-api" Oct 31 19:51:29.234982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.234964 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ee3f2a-332e-4c98-9b21-c76f3903b69f" containerName="payload-pre-processing" Oct 31 19:51:29.235225 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.235209 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1b9d6fd-7188-40bf-ae59-24c3f7ac8a7f" containerName="maas-api" Oct 31 19:51:29.235344 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.235329 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e8d77e7-9866-404f-83fc-1809898cec90" containerName="istio-proxy" Oct 31 19:51:29.235495 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.235452 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e8d77e7-9866-404f-83fc-1809898cec90" containerName="istio-proxy" Oct 31 19:51:29.235674 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.235663 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="69d2da79-5a06-4c5a-9185-58c9136a05cf" containerName="maas-api" Oct 31 19:51:29.235809 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.235781 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="73aa5edd-70dd-4bf1-b51c-ba31774baacd" containerName="istio-proxy" Oct 31 19:51:29.235904 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.235895 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fcfd21e-6616-4e59-a233-57f614e07231" containerName="revoke-keys" Oct 31 19:51:29.236123 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.236109 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4d3ccc1-b910-4e07-8d4a-8a55e19070a5" containerName="revoke-keys" Oct 31 19:51:29.243365 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.243325 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5" Oct 31 19:51:29.254802 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.254749 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c42f50-c084-4277-803e-37fa5e20764f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5\" (UID: \"02c42f50-c084-4277-803e-37fa5e20764f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5" Oct 31 19:51:29.255124 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.255099 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8jgq\" (UniqueName: \"kubernetes.io/projected/02c42f50-c084-4277-803e-37fa5e20764f-kube-api-access-r8jgq\") pod \"maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5\" (UID: \"02c42f50-c084-4277-803e-37fa5e20764f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5" Oct 31 19:51:29.255618 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.255579 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c42f50-c084-4277-803e-37fa5e20764f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5\" (UID: \"02c42f50-c084-4277-803e-37fa5e20764f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5" Oct 31 19:51:29.256929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.256894 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:51:29.257547 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.257530 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:51:29.290583 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.290550 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5"] Oct 31 19:51:29.357058 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.357012 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c42f50-c084-4277-803e-37fa5e20764f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5\" (UID: \"02c42f50-c084-4277-803e-37fa5e20764f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5" Oct 31 19:51:29.357290 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.357068 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8jgq\" (UniqueName: \"kubernetes.io/projected/02c42f50-c084-4277-803e-37fa5e20764f-kube-api-access-r8jgq\") pod \"maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5\" (UID: \"02c42f50-c084-4277-803e-37fa5e20764f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5" Oct 31 19:51:29.357452 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.357431 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c42f50-c084-4277-803e-37fa5e20764f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5\" (UID: \"02c42f50-c084-4277-803e-37fa5e20764f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5" Oct 31 19:51:29.358475 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.358445 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c42f50-c084-4277-803e-37fa5e20764f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5\" (UID: \"02c42f50-c084-4277-803e-37fa5e20764f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5" Oct 31 19:51:29.366258 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.366228 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c42f50-c084-4277-803e-37fa5e20764f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5\" (UID: \"02c42f50-c084-4277-803e-37fa5e20764f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5" Oct 31 19:51:29.371366 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.371337 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8jgq\" (UniqueName: \"kubernetes.io/projected/02c42f50-c084-4277-803e-37fa5e20764f-kube-api-access-r8jgq\") pod \"maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5\" (UID: \"02c42f50-c084-4277-803e-37fa5e20764f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5" Oct 31 19:51:29.571018 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.570983 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5" Oct 31 19:51:29.757526 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.757491 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69d2da79-5a06-4c5a-9185-58c9136a05cf" path="/var/lib/kubelet/pods/69d2da79-5a06-4c5a-9185-58c9136a05cf/volumes" Oct 31 19:51:29.869064 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:29.869023 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5"] Oct 31 19:51:29.870247 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:51:29.870212 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c42f50_c084_4277_803e_37fa5e20764f.slice/crio-bcf150b1c78503b65ab857c89e988220bad8ef17910a170398fa497f2dab7ce0 WatchSource:0}: Error finding container bcf150b1c78503b65ab857c89e988220bad8ef17910a170398fa497f2dab7ce0: Status 404 returned error can't find the container with id bcf150b1c78503b65ab857c89e988220bad8ef17910a170398fa497f2dab7ce0 Oct 31 19:51:30.211887 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:30.211851 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5" event={"ID":"02c42f50-c084-4277-803e-37fa5e20764f","Type":"ContainerStarted","Data":"f94c942614dc91bc71299e7af736e9afc6a921bd26ca15122ccdc10a26c1a5f1"} Oct 31 19:51:30.212070 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:30.211942 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5" event={"ID":"02c42f50-c084-4277-803e-37fa5e20764f","Type":"ContainerStarted","Data":"bcf150b1c78503b65ab857c89e988220bad8ef17910a170398fa497f2dab7ce0"} Oct 31 19:51:30.236407 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:30.236358 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5" podStartSLOduration=1.236344619 podStartE2EDuration="1.236344619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:51:30.232504386 +0000 UTC m=+1924.595324757" watchObservedRunningTime="2026-10-31 19:51:30.236344619 +0000 UTC m=+1924.599164991" Oct 31 19:51:31.226597 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:31.226556 4910 generic.go:356] "Generic (PLEG): container finished" podID="02c42f50-c084-4277-803e-37fa5e20764f" containerID="f94c942614dc91bc71299e7af736e9afc6a921bd26ca15122ccdc10a26c1a5f1" exitCode=0 Oct 31 19:51:31.226597 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:31.226585 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5" event={"ID":"02c42f50-c084-4277-803e-37fa5e20764f","Type":"ContainerDied","Data":"f94c942614dc91bc71299e7af736e9afc6a921bd26ca15122ccdc10a26c1a5f1"} Oct 31 19:51:32.437128 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:32.437098 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5" Oct 31 19:51:32.528823 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:32.528744 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02c42f50-c084-4277-803e-37fa5e20764f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c42f50-c084-4277-803e-37fa5e20764f-maas-api-xks-ca\") pod \"02c42f50-c084-4277-803e-37fa5e20764f\" (UID: \"02c42f50-c084-4277-803e-37fa5e20764f\") " Oct 31 19:51:32.529016 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:32.528823 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02c42f50-c084-4277-803e-37fa5e20764f-kube-api-access-r8jgq\" (UniqueName: \"kubernetes.io/projected/02c42f50-c084-4277-803e-37fa5e20764f-kube-api-access-r8jgq\") pod \"02c42f50-c084-4277-803e-37fa5e20764f\" (UID: \"02c42f50-c084-4277-803e-37fa5e20764f\") " Oct 31 19:51:32.529016 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:32.528900 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02c42f50-c084-4277-803e-37fa5e20764f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c42f50-c084-4277-803e-37fa5e20764f-maas-api-openshift-ca\") pod \"02c42f50-c084-4277-803e-37fa5e20764f\" (UID: \"02c42f50-c084-4277-803e-37fa5e20764f\") " Oct 31 19:51:32.529261 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:32.529234 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c42f50-c084-4277-803e-37fa5e20764f-maas-api-openshift-ca" pod "02c42f50-c084-4277-803e-37fa5e20764f" (UID: "02c42f50-c084-4277-803e-37fa5e20764f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:51:32.529544 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:32.529518 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02c42f50-c084-4277-803e-37fa5e20764f-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:32.531515 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:32.531488 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02c42f50-c084-4277-803e-37fa5e20764f-maas-api-xks-ca" pod "02c42f50-c084-4277-803e-37fa5e20764f" (UID: "02c42f50-c084-4277-803e-37fa5e20764f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:51:32.531617 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:32.531523 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c42f50-c084-4277-803e-37fa5e20764f-kube-api-access-r8jgq" pod "02c42f50-c084-4277-803e-37fa5e20764f" (UID: "02c42f50-c084-4277-803e-37fa5e20764f"). InnerVolumeSpecName "kube-api-access-r8jgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:32.630648 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:32.630613 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8jgq\" (UniqueName: \"kubernetes.io/projected/02c42f50-c084-4277-803e-37fa5e20764f-kube-api-access-r8jgq\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:32.630648 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:32.630645 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02c42f50-c084-4277-803e-37fa5e20764f-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:33.251458 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:33.251418 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5" event={"ID":"02c42f50-c084-4277-803e-37fa5e20764f","Type":"ContainerDied","Data":"bcf150b1c78503b65ab857c89e988220bad8ef17910a170398fa497f2dab7ce0"} Oct 31 19:51:33.251663 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:33.251476 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcf150b1c78503b65ab857c89e988220bad8ef17910a170398fa497f2dab7ce0" Oct 31 19:51:33.251663 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:33.251432 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5" Oct 31 19:51:33.790737 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:33.790703 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb" podUID="62d760d9-672c-4690-b5de-e8c6456c5431" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 19:51:33.809542 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:33.809516 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:51:33.952170 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:33.952136 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-token\" (UniqueName: \"kubernetes.io/projected/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-token\") pod \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " Oct 31 19:51:33.952392 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:33.952206 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-workload-socket\") pod \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " Oct 31 19:51:33.952392 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:33.952240 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c86e67dd-5de9-4213-b26d-fcbc3bd90353-kube-api-access-vswhv\" (UniqueName: \"kubernetes.io/projected/c86e67dd-5de9-4213-b26d-fcbc3bd90353-kube-api-access-vswhv\") pod \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " Oct 31 19:51:33.952392 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:33.952274 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istiod-ca-cert\") pod \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " Oct 31 19:51:33.952392 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:33.952386 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-podinfo\") pod \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " Oct 31 19:51:33.953128 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:33.952435 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-data\") pod \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " Oct 31 19:51:33.953128 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:33.952508 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-credential-socket\") pod \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " Oct 31 19:51:33.953128 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:33.952617 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-workload-certs\") pod \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " Oct 31 19:51:33.953128 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:33.952762 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-envoy\") pod \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\" (UID: \"c86e67dd-5de9-4213-b26d-fcbc3bd90353\") " Oct 31 19:51:33.953128 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:33.953034 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istiod-ca-cert" pod "c86e67dd-5de9-4213-b26d-fcbc3bd90353" (UID: "c86e67dd-5de9-4213-b26d-fcbc3bd90353"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:51:33.953448 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:33.953262 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-data" pod "c86e67dd-5de9-4213-b26d-fcbc3bd90353" (UID: "c86e67dd-5de9-4213-b26d-fcbc3bd90353"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:33.953659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:33.953634 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:33.953659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:33.953657 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:33.954070 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:33.953722 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-credential-socket" pod "c86e67dd-5de9-4213-b26d-fcbc3bd90353" (UID: "c86e67dd-5de9-4213-b26d-fcbc3bd90353"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:33.954185 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:33.954166 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-workload-certs" pod "c86e67dd-5de9-4213-b26d-fcbc3bd90353" (UID: "c86e67dd-5de9-4213-b26d-fcbc3bd90353"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:33.954688 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:33.954655 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-workload-socket" pod "c86e67dd-5de9-4213-b26d-fcbc3bd90353" (UID: "c86e67dd-5de9-4213-b26d-fcbc3bd90353"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:33.956011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:33.955851 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-envoy" pod "c86e67dd-5de9-4213-b26d-fcbc3bd90353" (UID: "c86e67dd-5de9-4213-b26d-fcbc3bd90353"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:33.956011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:33.955978 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-token" pod "c86e67dd-5de9-4213-b26d-fcbc3bd90353" (UID: "c86e67dd-5de9-4213-b26d-fcbc3bd90353"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:33.956978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:33.956949 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-podinfo" pod "c86e67dd-5de9-4213-b26d-fcbc3bd90353" (UID: "c86e67dd-5de9-4213-b26d-fcbc3bd90353"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:51:33.957670 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:33.957639 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86e67dd-5de9-4213-b26d-fcbc3bd90353-kube-api-access-vswhv" pod "c86e67dd-5de9-4213-b26d-fcbc3bd90353" (UID: "c86e67dd-5de9-4213-b26d-fcbc3bd90353"). InnerVolumeSpecName "kube-api-access-vswhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:34.074537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:34.064263 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:34.074537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:34.064301 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:34.074537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:34.064318 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:34.074537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:34.064334 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vswhv\" (UniqueName: \"kubernetes.io/projected/c86e67dd-5de9-4213-b26d-fcbc3bd90353-kube-api-access-vswhv\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:34.074537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:34.064350 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c86e67dd-5de9-4213-b26d-fcbc3bd90353-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:34.074537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:34.064365 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:34.074537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:34.064380 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c86e67dd-5de9-4213-b26d-fcbc3bd90353-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:34.264763 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:34.264734 4910 generic.go:356] "Generic (PLEG): container finished" podID="c86e67dd-5de9-4213-b26d-fcbc3bd90353" containerID="418d0eab141ae1e42b12d7018a9cbf0f5f2f5afaa4c005214239ea3eedec345e" exitCode=0 Oct 31 19:51:34.265040 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:34.264966 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" event={"ID":"c86e67dd-5de9-4213-b26d-fcbc3bd90353","Type":"ContainerDied","Data":"418d0eab141ae1e42b12d7018a9cbf0f5f2f5afaa4c005214239ea3eedec345e"} Oct 31 19:51:34.265130 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:34.265049 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" event={"ID":"c86e67dd-5de9-4213-b26d-fcbc3bd90353","Type":"ContainerDied","Data":"87f19639741372e2c282ea8c1536146ca6ea130f76cd25d5c2370ee5d5ea4160"} Oct 31 19:51:34.265130 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:34.265078 4910 scope.go:122] "RemoveContainer" containerID="418d0eab141ae1e42b12d7018a9cbf0f5f2f5afaa4c005214239ea3eedec345e" Oct 31 19:51:34.265130 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:34.265100 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb" Oct 31 19:51:34.280192 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:34.280166 4910 scope.go:122] "RemoveContainer" containerID="61e8372de968812a78948b9889ca8ea08d70b5f79321f7283cf56b354d18f00a" Oct 31 19:51:34.291513 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:34.291496 4910 scope.go:122] "RemoveContainer" containerID="418d0eab141ae1e42b12d7018a9cbf0f5f2f5afaa4c005214239ea3eedec345e" Oct 31 19:51:34.291974 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:51:34.291949 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"418d0eab141ae1e42b12d7018a9cbf0f5f2f5afaa4c005214239ea3eedec345e\": container with ID starting with 418d0eab141ae1e42b12d7018a9cbf0f5f2f5afaa4c005214239ea3eedec345e not found: ID does not exist" containerID="418d0eab141ae1e42b12d7018a9cbf0f5f2f5afaa4c005214239ea3eedec345e" Oct 31 19:51:34.292092 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:34.291987 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"418d0eab141ae1e42b12d7018a9cbf0f5f2f5afaa4c005214239ea3eedec345e"} err="failed to get container status \"418d0eab141ae1e42b12d7018a9cbf0f5f2f5afaa4c005214239ea3eedec345e\": rpc error: code = NotFound desc = could not find container \"418d0eab141ae1e42b12d7018a9cbf0f5f2f5afaa4c005214239ea3eedec345e\": container with ID starting with 418d0eab141ae1e42b12d7018a9cbf0f5f2f5afaa4c005214239ea3eedec345e not found: ID does not exist" Oct 31 19:51:34.292092 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:34.292011 4910 scope.go:122] "RemoveContainer" containerID="61e8372de968812a78948b9889ca8ea08d70b5f79321f7283cf56b354d18f00a" Oct 31 19:51:34.292749 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:51:34.292567 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61e8372de968812a78948b9889ca8ea08d70b5f79321f7283cf56b354d18f00a\": container with ID starting with 61e8372de968812a78948b9889ca8ea08d70b5f79321f7283cf56b354d18f00a not found: ID does not exist" containerID="61e8372de968812a78948b9889ca8ea08d70b5f79321f7283cf56b354d18f00a" Oct 31 19:51:34.292749 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:34.292607 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61e8372de968812a78948b9889ca8ea08d70b5f79321f7283cf56b354d18f00a"} err="failed to get container status \"61e8372de968812a78948b9889ca8ea08d70b5f79321f7283cf56b354d18f00a\": rpc error: code = NotFound desc = could not find container \"61e8372de968812a78948b9889ca8ea08d70b5f79321f7283cf56b354d18f00a\": container with ID starting with 61e8372de968812a78948b9889ca8ea08d70b5f79321f7283cf56b354d18f00a not found: ID does not exist" Oct 31 19:51:34.298568 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:34.298543 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb"] Oct 31 19:51:34.302981 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:34.302959 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1e8cf0-openshift-default-68f469b489-w6rnb"] Oct 31 19:51:35.758449 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:35.758412 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c86e67dd-5de9-4213-b26d-fcbc3bd90353" path="/var/lib/kubelet/pods/c86e67dd-5de9-4213-b26d-fcbc3bd90353/volumes" Oct 31 19:51:40.421447 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:40.421417 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb" Oct 31 19:51:40.463315 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:40.463275 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62d760d9-672c-4690-b5de-e8c6456c5431-kube-api-access-5gnkp\" (UniqueName: \"kubernetes.io/projected/62d760d9-672c-4690-b5de-e8c6456c5431-kube-api-access-5gnkp\") pod \"62d760d9-672c-4690-b5de-e8c6456c5431\" (UID: \"62d760d9-672c-4690-b5de-e8c6456c5431\") " Oct 31 19:51:40.463699 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:40.463670 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62d760d9-672c-4690-b5de-e8c6456c5431-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62d760d9-672c-4690-b5de-e8c6456c5431-plugins-config-volume\") pod \"62d760d9-672c-4690-b5de-e8c6456c5431\" (UID: \"62d760d9-672c-4690-b5de-e8c6456c5431\") " Oct 31 19:51:40.465852 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:40.465802 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d760d9-672c-4690-b5de-e8c6456c5431-plugins-config-volume" pod "62d760d9-672c-4690-b5de-e8c6456c5431" (UID: "62d760d9-672c-4690-b5de-e8c6456c5431"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:51:40.473017 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:40.471630 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d760d9-672c-4690-b5de-e8c6456c5431-kube-api-access-5gnkp" pod "62d760d9-672c-4690-b5de-e8c6456c5431" (UID: "62d760d9-672c-4690-b5de-e8c6456c5431"). InnerVolumeSpecName "kube-api-access-5gnkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:40.477145 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:40.476821 4910 generic.go:356] "Generic (PLEG): container finished" podID="62d760d9-672c-4690-b5de-e8c6456c5431" containerID="1f8427e1acc867c2d79b638d77272c277c74ca79b003de08f87083d6aff05c9e" exitCode=0 Oct 31 19:51:40.477145 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:40.476886 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb" Oct 31 19:51:40.477145 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:40.476905 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb" event={"ID":"62d760d9-672c-4690-b5de-e8c6456c5431","Type":"ContainerDied","Data":"1f8427e1acc867c2d79b638d77272c277c74ca79b003de08f87083d6aff05c9e"} Oct 31 19:51:40.477145 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:40.476979 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb" event={"ID":"62d760d9-672c-4690-b5de-e8c6456c5431","Type":"ContainerDied","Data":"9ac3dec9a187f8558b06685c2ac9ac9874ba45cba0316954d70af001bdebcfa4"} Oct 31 19:51:40.477145 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:40.477009 4910 scope.go:122] "RemoveContainer" containerID="1f8427e1acc867c2d79b638d77272c277c74ca79b003de08f87083d6aff05c9e" Oct 31 19:51:40.530943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:40.530400 4910 scope.go:122] "RemoveContainer" containerID="1f8427e1acc867c2d79b638d77272c277c74ca79b003de08f87083d6aff05c9e" Oct 31 19:51:40.530943 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:51:40.530799 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f8427e1acc867c2d79b638d77272c277c74ca79b003de08f87083d6aff05c9e\": container with ID starting with 1f8427e1acc867c2d79b638d77272c277c74ca79b003de08f87083d6aff05c9e not found: ID does not exist" containerID="1f8427e1acc867c2d79b638d77272c277c74ca79b003de08f87083d6aff05c9e" Oct 31 19:51:40.530943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:40.530833 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f8427e1acc867c2d79b638d77272c277c74ca79b003de08f87083d6aff05c9e"} err="failed to get container status \"1f8427e1acc867c2d79b638d77272c277c74ca79b003de08f87083d6aff05c9e\": rpc error: code = NotFound desc = could not find container \"1f8427e1acc867c2d79b638d77272c277c74ca79b003de08f87083d6aff05c9e\": container with ID starting with 1f8427e1acc867c2d79b638d77272c277c74ca79b003de08f87083d6aff05c9e not found: ID does not exist" Oct 31 19:51:40.557217 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:40.557179 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb"] Oct 31 19:51:40.561257 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:40.561232 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1e8cf0-69f79d459f-4lmcb"] Oct 31 19:51:40.566923 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:40.566885 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gnkp\" (UniqueName: \"kubernetes.io/projected/62d760d9-672c-4690-b5de-e8c6456c5431-kube-api-access-5gnkp\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:40.567035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:40.566925 4910 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62d760d9-672c-4690-b5de-e8c6456c5431-plugins-config-volume\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:41.791118 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:41.788865 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62d760d9-672c-4690-b5de-e8c6456c5431" path="/var/lib/kubelet/pods/62d760d9-672c-4690-b5de-e8c6456c5431/volumes" Oct 31 19:51:42.717637 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:51:42.717594 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:51:43.124308 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:43.124217 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5"] Oct 31 19:51:43.137953 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:43.127568 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8dcfa8-f2pz5"] Oct 31 19:51:43.758837 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:43.758802 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02c42f50-c084-4277-803e-37fa5e20764f" path="/var/lib/kubelet/pods/02c42f50-c084-4277-803e-37fa5e20764f/volumes" Oct 31 19:51:51.359992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:51.359945 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8772c052-zplbh"] Oct 31 19:51:51.360953 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:51.360902 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c86e67dd-5de9-4213-b26d-fcbc3bd90353" containerName="istio-proxy" Oct 31 19:51:51.360953 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:51.360941 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c86e67dd-5de9-4213-b26d-fcbc3bd90353" containerName="istio-proxy" Oct 31 19:51:51.361089 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:51.360995 4910 cpu_manager.go:414] "containerMap: removing container" podUID="02c42f50-c084-4277-803e-37fa5e20764f" containerName="revoke-keys" Oct 31 19:51:51.361089 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:51.361003 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c42f50-c084-4277-803e-37fa5e20764f" containerName="revoke-keys" Oct 31 19:51:51.361089 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:51.361041 4910 cpu_manager.go:414] "containerMap: removing container" podUID="62d760d9-672c-4690-b5de-e8c6456c5431" containerName="payload-pre-processing" Oct 31 19:51:51.361089 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:51.361049 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62d760d9-672c-4690-b5de-e8c6456c5431" containerName="payload-pre-processing" Oct 31 19:51:51.361241 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:51.361228 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c42f50-c084-4277-803e-37fa5e20764f" containerName="revoke-keys" Oct 31 19:51:51.361278 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:51.361264 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="62d760d9-672c-4690-b5de-e8c6456c5431" containerName="payload-pre-processing" Oct 31 19:51:51.361313 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:51.361278 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c86e67dd-5de9-4213-b26d-fcbc3bd90353" containerName="istio-proxy" Oct 31 19:51:51.361313 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:51.361288 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c86e67dd-5de9-4213-b26d-fcbc3bd90353" containerName="istio-proxy" Oct 31 19:51:51.365835 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:51.365791 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8772c052-zplbh" Oct 31 19:51:51.376587 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:51.376304 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:51:51.377101 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:51.377077 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:51:51.377362 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:51.377338 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8772c052-zplbh"] Oct 31 19:51:51.411649 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:51.411614 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghdq4\" (UniqueName: \"kubernetes.io/projected/cbe6143b-0623-4419-a8eb-5a5047ae149a-kube-api-access-ghdq4\") pod \"maas-api-revoke-sub-8772c052-zplbh\" (UID: \"cbe6143b-0623-4419-a8eb-5a5047ae149a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8772c052-zplbh" Oct 31 19:51:51.411830 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:51.411798 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe6143b-0623-4419-a8eb-5a5047ae149a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8772c052-zplbh\" (UID: \"cbe6143b-0623-4419-a8eb-5a5047ae149a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8772c052-zplbh" Oct 31 19:51:51.411898 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:51.411849 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe6143b-0623-4419-a8eb-5a5047ae149a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8772c052-zplbh\" (UID: \"cbe6143b-0623-4419-a8eb-5a5047ae149a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8772c052-zplbh" Oct 31 19:51:51.513058 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:51.513018 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe6143b-0623-4419-a8eb-5a5047ae149a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8772c052-zplbh\" (UID: \"cbe6143b-0623-4419-a8eb-5a5047ae149a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8772c052-zplbh" Oct 31 19:51:51.513232 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:51.513066 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe6143b-0623-4419-a8eb-5a5047ae149a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8772c052-zplbh\" (UID: \"cbe6143b-0623-4419-a8eb-5a5047ae149a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8772c052-zplbh" Oct 31 19:51:51.513346 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:51.513329 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghdq4\" (UniqueName: \"kubernetes.io/projected/cbe6143b-0623-4419-a8eb-5a5047ae149a-kube-api-access-ghdq4\") pod \"maas-api-revoke-sub-8772c052-zplbh\" (UID: \"cbe6143b-0623-4419-a8eb-5a5047ae149a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8772c052-zplbh" Oct 31 19:51:51.513671 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:51.513650 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe6143b-0623-4419-a8eb-5a5047ae149a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8772c052-zplbh\" (UID: \"cbe6143b-0623-4419-a8eb-5a5047ae149a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8772c052-zplbh" Oct 31 19:51:51.515523 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:51.515501 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe6143b-0623-4419-a8eb-5a5047ae149a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8772c052-zplbh\" (UID: \"cbe6143b-0623-4419-a8eb-5a5047ae149a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8772c052-zplbh" Oct 31 19:51:51.526476 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:51.526449 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghdq4\" (UniqueName: \"kubernetes.io/projected/cbe6143b-0623-4419-a8eb-5a5047ae149a-kube-api-access-ghdq4\") pod \"maas-api-revoke-sub-8772c052-zplbh\" (UID: \"cbe6143b-0623-4419-a8eb-5a5047ae149a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8772c052-zplbh" Oct 31 19:51:51.699511 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:51.699419 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8772c052-zplbh" Oct 31 19:51:52.038140 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:52.038023 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk"] Oct 31 19:51:52.040379 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:52.038968 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" podUID="67b3d8c4-ff06-44a2-a9af-d945b5d238ae" containerName="istio-proxy" containerID="cri-o://eef4eefc3606e9183c784521515adbd1b503e105642ae42b0f2d12ed5ea0cee1" gracePeriod=30 Oct 31 19:51:52.042536 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:52.042504 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz"] Oct 31 19:51:52.042752 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:52.042727 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" podUID="6da8e70c-0b96-48e1-92cb-01ca01434b98" containerName="istio-proxy" containerID="cri-o://eb9890bd5fc2b594175fc4dd1dfd6dfda00142fd531d62457d189ad341193e1b" gracePeriod=30 Oct 31 19:51:52.064424 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:52.064375 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8772c052-zplbh"] Oct 31 19:51:52.889801 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:52.889613 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8772c052-zplbh" event={"ID":"cbe6143b-0623-4419-a8eb-5a5047ae149a","Type":"ContainerStarted","Data":"9b3c751d1209223707d9eb8ea8a94a3c045e6b33dc26a46cca8433697ae46275"} Oct 31 19:51:52.889801 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:52.889685 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8772c052-zplbh" event={"ID":"cbe6143b-0623-4419-a8eb-5a5047ae149a","Type":"ContainerStarted","Data":"2e392b285f68f23f35deddc3a42b6bb8061f48d87ebdc29aa20a4a40b0c25cbf"} Oct 31 19:51:53.910269 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:53.910225 4910 generic.go:356] "Generic (PLEG): container finished" podID="cbe6143b-0623-4419-a8eb-5a5047ae149a" containerID="9b3c751d1209223707d9eb8ea8a94a3c045e6b33dc26a46cca8433697ae46275" exitCode=0 Oct 31 19:51:53.910777 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:53.910510 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8772c052-zplbh" event={"ID":"cbe6143b-0623-4419-a8eb-5a5047ae149a","Type":"ContainerDied","Data":"9b3c751d1209223707d9eb8ea8a94a3c045e6b33dc26a46cca8433697ae46275"} Oct 31 19:51:54.722139 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:54.722103 4910 patch_prober.go:29] interesting pod/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:51:54.722322 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:54.722168 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" podUID="6da8e70c-0b96-48e1-92cb-01ca01434b98" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:51:55.115194 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:55.115159 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8772c052-zplbh" Oct 31 19:51:55.246128 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:55.246034 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbe6143b-0623-4419-a8eb-5a5047ae149a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe6143b-0623-4419-a8eb-5a5047ae149a-maas-api-openshift-ca\") pod \"cbe6143b-0623-4419-a8eb-5a5047ae149a\" (UID: \"cbe6143b-0623-4419-a8eb-5a5047ae149a\") " Oct 31 19:51:55.246128 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:55.246125 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe6143b-0623-4419-a8eb-5a5047ae149a-kube-api-access-ghdq4\" (UniqueName: \"kubernetes.io/projected/cbe6143b-0623-4419-a8eb-5a5047ae149a-kube-api-access-ghdq4\") pod \"cbe6143b-0623-4419-a8eb-5a5047ae149a\" (UID: \"cbe6143b-0623-4419-a8eb-5a5047ae149a\") " Oct 31 19:51:55.246397 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:55.246376 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbe6143b-0623-4419-a8eb-5a5047ae149a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe6143b-0623-4419-a8eb-5a5047ae149a-maas-api-xks-ca\") pod \"cbe6143b-0623-4419-a8eb-5a5047ae149a\" (UID: \"cbe6143b-0623-4419-a8eb-5a5047ae149a\") " Oct 31 19:51:55.246492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:55.246458 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbe6143b-0623-4419-a8eb-5a5047ae149a-maas-api-openshift-ca" pod "cbe6143b-0623-4419-a8eb-5a5047ae149a" (UID: "cbe6143b-0623-4419-a8eb-5a5047ae149a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:51:55.247528 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:55.247093 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe6143b-0623-4419-a8eb-5a5047ae149a-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:55.248978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:55.248938 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe6143b-0623-4419-a8eb-5a5047ae149a-kube-api-access-ghdq4" pod "cbe6143b-0623-4419-a8eb-5a5047ae149a" (UID: "cbe6143b-0623-4419-a8eb-5a5047ae149a"). InnerVolumeSpecName "kube-api-access-ghdq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:55.249204 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:55.249173 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbe6143b-0623-4419-a8eb-5a5047ae149a-maas-api-xks-ca" pod "cbe6143b-0623-4419-a8eb-5a5047ae149a" (UID: "cbe6143b-0623-4419-a8eb-5a5047ae149a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:51:55.348591 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:55.348547 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe6143b-0623-4419-a8eb-5a5047ae149a-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:55.348591 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:55.348579 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghdq4\" (UniqueName: \"kubernetes.io/projected/cbe6143b-0623-4419-a8eb-5a5047ae149a-kube-api-access-ghdq4\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:55.928595 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:55.928563 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8772c052-zplbh" Oct 31 19:51:55.928772 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:55.928597 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8772c052-zplbh" event={"ID":"cbe6143b-0623-4419-a8eb-5a5047ae149a","Type":"ContainerDied","Data":"2e392b285f68f23f35deddc3a42b6bb8061f48d87ebdc29aa20a4a40b0c25cbf"} Oct 31 19:51:55.928772 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:55.928648 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e392b285f68f23f35deddc3a42b6bb8061f48d87ebdc29aa20a4a40b0c25cbf" Oct 31 19:51:57.474010 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.473963 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:51:57.478287 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.478261 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:51:57.607773 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.607737 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-workload-socket\") pod \"6da8e70c-0b96-48e1-92cb-01ca01434b98\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " Oct 31 19:51:57.608015 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.607785 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6da8e70c-0b96-48e1-92cb-01ca01434b98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da8e70c-0b96-48e1-92cb-01ca01434b98-istiod-ca-cert\") pod \"6da8e70c-0b96-48e1-92cb-01ca01434b98\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " Oct 31 19:51:57.608015 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.607818 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-workload-certs\") pod \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " Oct 31 19:51:57.608015 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.607852 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-credential-socket\") pod \"6da8e70c-0b96-48e1-92cb-01ca01434b98\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " Oct 31 19:51:57.608015 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.607991 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-podinfo\") pod \"6da8e70c-0b96-48e1-92cb-01ca01434b98\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " Oct 31 19:51:57.608264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.608045 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-workload-socket\") pod \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " Oct 31 19:51:57.608264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.608093 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-workload-socket" pod "6da8e70c-0b96-48e1-92cb-01ca01434b98" (UID: "6da8e70c-0b96-48e1-92cb-01ca01434b98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:57.608264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.608157 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-credential-socket\") pod \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " Oct 31 19:51:57.608264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.608146 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-workload-certs" pod "67b3d8c4-ff06-44a2-a9af-d945b5d238ae" (UID: "67b3d8c4-ff06-44a2-a9af-d945b5d238ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:57.608264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.608222 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-credential-socket" pod "6da8e70c-0b96-48e1-92cb-01ca01434b98" (UID: "6da8e70c-0b96-48e1-92cb-01ca01434b98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:57.608264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.608238 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da8e70c-0b96-48e1-92cb-01ca01434b98-kube-api-access-jwqm9\" (UniqueName: \"kubernetes.io/projected/6da8e70c-0b96-48e1-92cb-01ca01434b98-kube-api-access-jwqm9\") pod \"6da8e70c-0b96-48e1-92cb-01ca01434b98\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " Oct 31 19:51:57.608582 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.608274 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-workload-certs\") pod \"6da8e70c-0b96-48e1-92cb-01ca01434b98\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " Oct 31 19:51:57.608582 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.608304 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da8e70c-0b96-48e1-92cb-01ca01434b98-istiod-ca-cert" pod "6da8e70c-0b96-48e1-92cb-01ca01434b98" (UID: "6da8e70c-0b96-48e1-92cb-01ca01434b98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:51:57.608582 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.608359 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-podinfo\") pod \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " Oct 31 19:51:57.608582 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.608396 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-envoy\") pod \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " Oct 31 19:51:57.608582 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.608443 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-data\") pod \"6da8e70c-0b96-48e1-92cb-01ca01434b98\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " Oct 31 19:51:57.608582 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.608426 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-workload-socket" pod "67b3d8c4-ff06-44a2-a9af-d945b5d238ae" (UID: "67b3d8c4-ff06-44a2-a9af-d945b5d238ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:57.608582 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.608491 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-credential-socket" pod "67b3d8c4-ff06-44a2-a9af-d945b5d238ae" (UID: "67b3d8c4-ff06-44a2-a9af-d945b5d238ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:57.608582 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.608499 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-data\") pod \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " Oct 31 19:51:57.608582 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.608568 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-workload-certs" pod "6da8e70c-0b96-48e1-92cb-01ca01434b98" (UID: "6da8e70c-0b96-48e1-92cb-01ca01434b98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:57.608582 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.608577 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-envoy\") pod \"6da8e70c-0b96-48e1-92cb-01ca01434b98\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " Oct 31 19:51:57.609225 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.608627 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-token\" (UniqueName: \"kubernetes.io/projected/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-token\") pod \"6da8e70c-0b96-48e1-92cb-01ca01434b98\" (UID: \"6da8e70c-0b96-48e1-92cb-01ca01434b98\") " Oct 31 19:51:57.609225 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.608698 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-kube-api-access-6bx9m\" (UniqueName: \"kubernetes.io/projected/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-kube-api-access-6bx9m\") pod \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " Oct 31 19:51:57.609225 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.608746 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-data" pod "67b3d8c4-ff06-44a2-a9af-d945b5d238ae" (UID: "67b3d8c4-ff06-44a2-a9af-d945b5d238ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:57.609225 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.608775 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istiod-ca-cert\") pod \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " Oct 31 19:51:57.609225 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.608837 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-token\" (UniqueName: \"kubernetes.io/projected/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-token\") pod \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\" (UID: \"67b3d8c4-ff06-44a2-a9af-d945b5d238ae\") " Oct 31 19:51:57.609225 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.608977 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-data" pod "6da8e70c-0b96-48e1-92cb-01ca01434b98" (UID: "6da8e70c-0b96-48e1-92cb-01ca01434b98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:57.609733 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.609707 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:57.609825 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.609739 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:57.609825 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.609754 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:57.609825 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.609769 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da8e70c-0b96-48e1-92cb-01ca01434b98-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:57.609825 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.609782 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:57.609825 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.609795 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:57.609825 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.609808 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:57.609825 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.609822 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:57.610259 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.609836 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:57.611168 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.610906 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-podinfo" pod "67b3d8c4-ff06-44a2-a9af-d945b5d238ae" (UID: "67b3d8c4-ff06-44a2-a9af-d945b5d238ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:51:57.611168 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.610997 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da8e70c-0b96-48e1-92cb-01ca01434b98-kube-api-access-jwqm9" pod "6da8e70c-0b96-48e1-92cb-01ca01434b98" (UID: "6da8e70c-0b96-48e1-92cb-01ca01434b98"). InnerVolumeSpecName "kube-api-access-jwqm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:57.611168 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.611008 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-podinfo" pod "6da8e70c-0b96-48e1-92cb-01ca01434b98" (UID: "6da8e70c-0b96-48e1-92cb-01ca01434b98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:51:57.611168 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.611019 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-envoy" pod "67b3d8c4-ff06-44a2-a9af-d945b5d238ae" (UID: "67b3d8c4-ff06-44a2-a9af-d945b5d238ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:57.611437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.611340 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-kube-api-access-6bx9m" pod "67b3d8c4-ff06-44a2-a9af-d945b5d238ae" (UID: "67b3d8c4-ff06-44a2-a9af-d945b5d238ae"). InnerVolumeSpecName "kube-api-access-6bx9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:57.611654 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.611628 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istiod-ca-cert" pod "67b3d8c4-ff06-44a2-a9af-d945b5d238ae" (UID: "67b3d8c4-ff06-44a2-a9af-d945b5d238ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:51:57.612228 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.612192 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-token" pod "6da8e70c-0b96-48e1-92cb-01ca01434b98" (UID: "6da8e70c-0b96-48e1-92cb-01ca01434b98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:57.612228 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.612216 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-token" pod "67b3d8c4-ff06-44a2-a9af-d945b5d238ae" (UID: "67b3d8c4-ff06-44a2-a9af-d945b5d238ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:51:57.612547 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.612517 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-envoy" pod "6da8e70c-0b96-48e1-92cb-01ca01434b98" (UID: "6da8e70c-0b96-48e1-92cb-01ca01434b98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:51:57.711778 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.711691 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:57.711778 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.711726 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:57.711778 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.711736 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:57.711778 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.711745 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:57.711778 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.711754 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bx9m\" (UniqueName: \"kubernetes.io/projected/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-kube-api-access-6bx9m\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:57.711778 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.711768 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:57.711778 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.711780 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67b3d8c4-ff06-44a2-a9af-d945b5d238ae-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:57.712172 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.711794 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da8e70c-0b96-48e1-92cb-01ca01434b98-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:57.712172 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.711807 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwqm9\" (UniqueName: \"kubernetes.io/projected/6da8e70c-0b96-48e1-92cb-01ca01434b98-kube-api-access-jwqm9\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:51:57.945733 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.945693 4910 generic.go:356] "Generic (PLEG): container finished" podID="6da8e70c-0b96-48e1-92cb-01ca01434b98" containerID="eb9890bd5fc2b594175fc4dd1dfd6dfda00142fd531d62457d189ad341193e1b" exitCode=0 Oct 31 19:51:57.945944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.945758 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" Oct 31 19:51:57.945944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.945770 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" event={"ID":"6da8e70c-0b96-48e1-92cb-01ca01434b98","Type":"ContainerDied","Data":"eb9890bd5fc2b594175fc4dd1dfd6dfda00142fd531d62457d189ad341193e1b"} Oct 31 19:51:57.945944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.945850 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz" event={"ID":"6da8e70c-0b96-48e1-92cb-01ca01434b98","Type":"ContainerDied","Data":"341a7b4dbc1eb682d80d084fa215a42ed6ca05838f9acf34718c21e00d8f337c"} Oct 31 19:51:57.945944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.945874 4910 scope.go:122] "RemoveContainer" containerID="eb9890bd5fc2b594175fc4dd1dfd6dfda00142fd531d62457d189ad341193e1b" Oct 31 19:51:57.947660 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.947639 4910 generic.go:356] "Generic (PLEG): container finished" podID="67b3d8c4-ff06-44a2-a9af-d945b5d238ae" containerID="eef4eefc3606e9183c784521515adbd1b503e105642ae42b0f2d12ed5ea0cee1" exitCode=0 Oct 31 19:51:57.947769 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.947685 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" Oct 31 19:51:57.947769 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.947724 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" event={"ID":"67b3d8c4-ff06-44a2-a9af-d945b5d238ae","Type":"ContainerDied","Data":"eef4eefc3606e9183c784521515adbd1b503e105642ae42b0f2d12ed5ea0cee1"} Oct 31 19:51:57.947769 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.947763 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk" event={"ID":"67b3d8c4-ff06-44a2-a9af-d945b5d238ae","Type":"ContainerDied","Data":"83b185de930718276d8a6cf5d6c9b76d2e8d58eb5d72d484b7a9acba33ed88de"} Oct 31 19:51:57.960803 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.960781 4910 scope.go:122] "RemoveContainer" containerID="33a0f003beabc7d5156e32eb35fff9331b705e3c20a2f984b0a1ee0e9d068749" Oct 31 19:51:57.972002 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.971931 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz"] Oct 31 19:51:57.974057 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.974035 4910 scope.go:122] "RemoveContainer" containerID="eb9890bd5fc2b594175fc4dd1dfd6dfda00142fd531d62457d189ad341193e1b" Oct 31 19:51:57.974365 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:51:57.974341 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb9890bd5fc2b594175fc4dd1dfd6dfda00142fd531d62457d189ad341193e1b\": container with ID starting with eb9890bd5fc2b594175fc4dd1dfd6dfda00142fd531d62457d189ad341193e1b not found: ID does not exist" containerID="eb9890bd5fc2b594175fc4dd1dfd6dfda00142fd531d62457d189ad341193e1b" Oct 31 19:51:57.974464 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.974373 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb9890bd5fc2b594175fc4dd1dfd6dfda00142fd531d62457d189ad341193e1b"} err="failed to get container status \"eb9890bd5fc2b594175fc4dd1dfd6dfda00142fd531d62457d189ad341193e1b\": rpc error: code = NotFound desc = could not find container \"eb9890bd5fc2b594175fc4dd1dfd6dfda00142fd531d62457d189ad341193e1b\": container with ID starting with eb9890bd5fc2b594175fc4dd1dfd6dfda00142fd531d62457d189ad341193e1b not found: ID does not exist" Oct 31 19:51:57.974464 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.974389 4910 scope.go:122] "RemoveContainer" containerID="33a0f003beabc7d5156e32eb35fff9331b705e3c20a2f984b0a1ee0e9d068749" Oct 31 19:51:57.974679 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:51:57.974657 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a0f003beabc7d5156e32eb35fff9331b705e3c20a2f984b0a1ee0e9d068749\": container with ID starting with 33a0f003beabc7d5156e32eb35fff9331b705e3c20a2f984b0a1ee0e9d068749 not found: ID does not exist" containerID="33a0f003beabc7d5156e32eb35fff9331b705e3c20a2f984b0a1ee0e9d068749" Oct 31 19:51:57.974745 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.974693 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a0f003beabc7d5156e32eb35fff9331b705e3c20a2f984b0a1ee0e9d068749"} err="failed to get container status \"33a0f003beabc7d5156e32eb35fff9331b705e3c20a2f984b0a1ee0e9d068749\": rpc error: code = NotFound desc = could not find container \"33a0f003beabc7d5156e32eb35fff9331b705e3c20a2f984b0a1ee0e9d068749\": container with ID starting with 33a0f003beabc7d5156e32eb35fff9331b705e3c20a2f984b0a1ee0e9d068749 not found: ID does not exist" Oct 31 19:51:57.974745 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.974716 4910 scope.go:122] "RemoveContainer" containerID="eef4eefc3606e9183c784521515adbd1b503e105642ae42b0f2d12ed5ea0cee1" Oct 31 19:51:57.977416 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.977398 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-9v4dz"] Oct 31 19:51:57.987790 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.987769 4910 scope.go:122] "RemoveContainer" containerID="eef4eefc3606e9183c784521515adbd1b503e105642ae42b0f2d12ed5ea0cee1" Oct 31 19:51:57.987938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.987902 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk"] Oct 31 19:51:57.988191 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:51:57.988162 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eef4eefc3606e9183c784521515adbd1b503e105642ae42b0f2d12ed5ea0cee1\": container with ID starting with eef4eefc3606e9183c784521515adbd1b503e105642ae42b0f2d12ed5ea0cee1 not found: ID does not exist" containerID="eef4eefc3606e9183c784521515adbd1b503e105642ae42b0f2d12ed5ea0cee1" Oct 31 19:51:57.988295 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.988195 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef4eefc3606e9183c784521515adbd1b503e105642ae42b0f2d12ed5ea0cee1"} err="failed to get container status \"eef4eefc3606e9183c784521515adbd1b503e105642ae42b0f2d12ed5ea0cee1\": rpc error: code = NotFound desc = could not find container \"eef4eefc3606e9183c784521515adbd1b503e105642ae42b0f2d12ed5ea0cee1\": container with ID starting with eef4eefc3606e9183c784521515adbd1b503e105642ae42b0f2d12ed5ea0cee1 not found: ID does not exist" Oct 31 19:51:57.993529 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:57.993508 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-z55xk"] Oct 31 19:51:59.757410 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:59.757370 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67b3d8c4-ff06-44a2-a9af-d945b5d238ae" path="/var/lib/kubelet/pods/67b3d8c4-ff06-44a2-a9af-d945b5d238ae/volumes" Oct 31 19:51:59.758122 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:51:59.758101 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6da8e70c-0b96-48e1-92cb-01ca01434b98" path="/var/lib/kubelet/pods/6da8e70c-0b96-48e1-92cb-01ca01434b98/volumes" Oct 31 19:52:06.442951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.442894 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c25c0405-s9h8f"] Oct 31 19:52:06.443579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.443563 4910 cpu_manager.go:414] "containerMap: removing container" podUID="67b3d8c4-ff06-44a2-a9af-d945b5d238ae" containerName="istio-proxy" Oct 31 19:52:06.443636 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.443580 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b3d8c4-ff06-44a2-a9af-d945b5d238ae" containerName="istio-proxy" Oct 31 19:52:06.443636 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.443603 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c86e67dd-5de9-4213-b26d-fcbc3bd90353" containerName="istio-proxy" Oct 31 19:52:06.443636 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.443609 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c86e67dd-5de9-4213-b26d-fcbc3bd90353" containerName="istio-proxy" Oct 31 19:52:06.443636 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.443628 4910 cpu_manager.go:414] "containerMap: removing container" podUID="6da8e70c-0b96-48e1-92cb-01ca01434b98" containerName="istio-proxy" Oct 31 19:52:06.443636 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.443633 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da8e70c-0b96-48e1-92cb-01ca01434b98" containerName="istio-proxy" Oct 31 19:52:06.443791 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.443645 4910 cpu_manager.go:414] "containerMap: removing container" podUID="6da8e70c-0b96-48e1-92cb-01ca01434b98" containerName="istio-proxy" Oct 31 19:52:06.443791 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.443651 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da8e70c-0b96-48e1-92cb-01ca01434b98" containerName="istio-proxy" Oct 31 19:52:06.443791 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.443669 4910 cpu_manager.go:414] "containerMap: removing container" podUID="cbe6143b-0623-4419-a8eb-5a5047ae149a" containerName="revoke-keys" Oct 31 19:52:06.443791 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.443674 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe6143b-0623-4419-a8eb-5a5047ae149a" containerName="revoke-keys" Oct 31 19:52:06.443791 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.443769 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b3d8c4-ff06-44a2-a9af-d945b5d238ae" containerName="istio-proxy" Oct 31 19:52:06.443791 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.443782 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da8e70c-0b96-48e1-92cb-01ca01434b98" containerName="istio-proxy" Oct 31 19:52:06.443791 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.443788 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da8e70c-0b96-48e1-92cb-01ca01434b98" containerName="istio-proxy" Oct 31 19:52:06.444062 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.443798 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe6143b-0623-4419-a8eb-5a5047ae149a" containerName="revoke-keys" Oct 31 19:52:06.449322 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.449303 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c25c0405-s9h8f" Oct 31 19:52:06.461670 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.461643 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c25c0405-s9h8f"] Oct 31 19:52:06.466387 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.466364 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:52:06.466585 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.466368 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:52:06.564056 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.564022 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bc602c5-0eb7-4b12-8821-961094ba097d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c25c0405-s9h8f\" (UID: \"5bc602c5-0eb7-4b12-8821-961094ba097d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c25c0405-s9h8f" Oct 31 19:52:06.564269 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.564091 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bc602c5-0eb7-4b12-8821-961094ba097d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c25c0405-s9h8f\" (UID: \"5bc602c5-0eb7-4b12-8821-961094ba097d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c25c0405-s9h8f" Oct 31 19:52:06.564269 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.564240 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbnh6\" (UniqueName: \"kubernetes.io/projected/5bc602c5-0eb7-4b12-8821-961094ba097d-kube-api-access-qbnh6\") pod \"maas-api-revoke-sub-c25c0405-s9h8f\" (UID: \"5bc602c5-0eb7-4b12-8821-961094ba097d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c25c0405-s9h8f" Oct 31 19:52:06.666243 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.666202 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbnh6\" (UniqueName: \"kubernetes.io/projected/5bc602c5-0eb7-4b12-8821-961094ba097d-kube-api-access-qbnh6\") pod \"maas-api-revoke-sub-c25c0405-s9h8f\" (UID: \"5bc602c5-0eb7-4b12-8821-961094ba097d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c25c0405-s9h8f" Oct 31 19:52:06.666455 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.666384 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bc602c5-0eb7-4b12-8821-961094ba097d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c25c0405-s9h8f\" (UID: \"5bc602c5-0eb7-4b12-8821-961094ba097d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c25c0405-s9h8f" Oct 31 19:52:06.666455 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.666429 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bc602c5-0eb7-4b12-8821-961094ba097d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c25c0405-s9h8f\" (UID: \"5bc602c5-0eb7-4b12-8821-961094ba097d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c25c0405-s9h8f" Oct 31 19:52:06.667255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.667231 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bc602c5-0eb7-4b12-8821-961094ba097d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c25c0405-s9h8f\" (UID: \"5bc602c5-0eb7-4b12-8821-961094ba097d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c25c0405-s9h8f" Oct 31 19:52:06.668878 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.668848 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bc602c5-0eb7-4b12-8821-961094ba097d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c25c0405-s9h8f\" (UID: \"5bc602c5-0eb7-4b12-8821-961094ba097d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c25c0405-s9h8f" Oct 31 19:52:06.698690 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.698621 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbnh6\" (UniqueName: \"kubernetes.io/projected/5bc602c5-0eb7-4b12-8821-961094ba097d-kube-api-access-qbnh6\") pod \"maas-api-revoke-sub-c25c0405-s9h8f\" (UID: \"5bc602c5-0eb7-4b12-8821-961094ba097d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c25c0405-s9h8f" Oct 31 19:52:06.767966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:06.767930 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c25c0405-s9h8f" Oct 31 19:52:07.063187 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:07.063105 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c25c0405-s9h8f"] Oct 31 19:52:07.063604 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:52:07.063577 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc602c5_0eb7_4b12_8821_961094ba097d.slice/crio-c842213bfc8522c71e498d4fdc14d5c9787f9befb3c04ad0043160249f0f452d WatchSource:0}: Error finding container c842213bfc8522c71e498d4fdc14d5c9787f9befb3c04ad0043160249f0f452d: Status 404 returned error can't find the container with id c842213bfc8522c71e498d4fdc14d5c9787f9befb3c04ad0043160249f0f452d Oct 31 19:52:07.078790 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:07.078763 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c25c0405-s9h8f" event={"ID":"5bc602c5-0eb7-4b12-8821-961094ba097d","Type":"ContainerStarted","Data":"c842213bfc8522c71e498d4fdc14d5c9787f9befb3c04ad0043160249f0f452d"} Oct 31 19:52:08.088734 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:08.088699 4910 generic.go:356] "Generic (PLEG): container finished" podID="5bc602c5-0eb7-4b12-8821-961094ba097d" containerID="ca655599226a631399cc6fb69d78b3a87ba9bfd7b1e5e6873d5df55405244375" exitCode=0 Oct 31 19:52:08.089205 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:08.088750 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c25c0405-s9h8f" event={"ID":"5bc602c5-0eb7-4b12-8821-961094ba097d","Type":"ContainerDied","Data":"ca655599226a631399cc6fb69d78b3a87ba9bfd7b1e5e6873d5df55405244375"} Oct 31 19:52:09.287214 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:09.287187 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c25c0405-s9h8f" Oct 31 19:52:09.413068 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:09.412988 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bc602c5-0eb7-4b12-8821-961094ba097d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bc602c5-0eb7-4b12-8821-961094ba097d-maas-api-xks-ca\") pod \"5bc602c5-0eb7-4b12-8821-961094ba097d\" (UID: \"5bc602c5-0eb7-4b12-8821-961094ba097d\") " Oct 31 19:52:09.413228 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:09.413147 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bc602c5-0eb7-4b12-8821-961094ba097d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bc602c5-0eb7-4b12-8821-961094ba097d-maas-api-openshift-ca\") pod \"5bc602c5-0eb7-4b12-8821-961094ba097d\" (UID: \"5bc602c5-0eb7-4b12-8821-961094ba097d\") " Oct 31 19:52:09.413228 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:09.413181 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bc602c5-0eb7-4b12-8821-961094ba097d-kube-api-access-qbnh6\" (UniqueName: \"kubernetes.io/projected/5bc602c5-0eb7-4b12-8821-961094ba097d-kube-api-access-qbnh6\") pod \"5bc602c5-0eb7-4b12-8821-961094ba097d\" (UID: \"5bc602c5-0eb7-4b12-8821-961094ba097d\") " Oct 31 19:52:09.413490 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:09.413464 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bc602c5-0eb7-4b12-8821-961094ba097d-maas-api-openshift-ca" pod "5bc602c5-0eb7-4b12-8821-961094ba097d" (UID: "5bc602c5-0eb7-4b12-8821-961094ba097d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:52:09.413849 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:09.413826 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bc602c5-0eb7-4b12-8821-961094ba097d-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:09.415095 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:09.415072 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bc602c5-0eb7-4b12-8821-961094ba097d-maas-api-xks-ca" pod "5bc602c5-0eb7-4b12-8821-961094ba097d" (UID: "5bc602c5-0eb7-4b12-8821-961094ba097d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:52:09.415296 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:09.415271 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc602c5-0eb7-4b12-8821-961094ba097d-kube-api-access-qbnh6" pod "5bc602c5-0eb7-4b12-8821-961094ba097d" (UID: "5bc602c5-0eb7-4b12-8821-961094ba097d"). InnerVolumeSpecName "kube-api-access-qbnh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:52:09.515530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:09.515498 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bc602c5-0eb7-4b12-8821-961094ba097d-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:09.515530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:09.515527 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbnh6\" (UniqueName: \"kubernetes.io/projected/5bc602c5-0eb7-4b12-8821-961094ba097d-kube-api-access-qbnh6\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:10.106127 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:10.106092 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c25c0405-s9h8f" Oct 31 19:52:10.106291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:10.106119 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c25c0405-s9h8f" event={"ID":"5bc602c5-0eb7-4b12-8821-961094ba097d","Type":"ContainerDied","Data":"c842213bfc8522c71e498d4fdc14d5c9787f9befb3c04ad0043160249f0f452d"} Oct 31 19:52:10.106291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:10.106181 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c842213bfc8522c71e498d4fdc14d5c9787f9befb3c04ad0043160249f0f452d" Oct 31 19:52:12.716808 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:52:12.716765 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:52:21.479850 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:21.479810 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t"] Oct 31 19:52:21.480372 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:21.480051 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" podUID="25d970e7-85b8-4057-8b20-5df84679b3ab" containerName="istio-proxy" containerID="cri-o://81d1f5dc97000d0eeb0a8510c17f6c12563b497d0dafe57db65ddacc09daa26d" gracePeriod=30 Oct 31 19:52:21.958474 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:21.957763 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-785d26-h8phb"] Oct 31 19:52:21.961302 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:21.959147 4910 cpu_manager.go:414] "containerMap: removing container" podUID="5bc602c5-0eb7-4b12-8821-961094ba097d" containerName="revoke-keys" Oct 31 19:52:21.961302 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:21.959174 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bc602c5-0eb7-4b12-8821-961094ba097d" containerName="revoke-keys" Oct 31 19:52:21.961302 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:21.959387 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bc602c5-0eb7-4b12-8821-961094ba097d" containerName="revoke-keys" Oct 31 19:52:21.963734 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:21.963714 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-785d26-h8phb" Oct 31 19:52:21.975674 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:21.975646 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-785d26-h8phb"] Oct 31 19:52:21.975936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:21.975889 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:52:21.976110 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:21.976093 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:52:22.154224 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:22.153927 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvk5r\" (UniqueName: \"kubernetes.io/projected/6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52-kube-api-access-xvk5r\") pod \"maas-api-revoke-keys-e2e-inf-a-785d26-h8phb\" (UID: \"6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-785d26-h8phb" Oct 31 19:52:22.154224 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:22.154222 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-785d26-h8phb\" (UID: \"6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-785d26-h8phb" Oct 31 19:52:22.154495 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:22.154256 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-785d26-h8phb\" (UID: \"6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-785d26-h8phb" Oct 31 19:52:22.256256 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:22.256163 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-785d26-h8phb\" (UID: \"6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-785d26-h8phb" Oct 31 19:52:22.256256 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:22.256216 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-785d26-h8phb\" (UID: \"6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-785d26-h8phb" Oct 31 19:52:22.256626 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:22.256328 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvk5r\" (UniqueName: \"kubernetes.io/projected/6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52-kube-api-access-xvk5r\") pod \"maas-api-revoke-keys-e2e-inf-a-785d26-h8phb\" (UID: \"6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-785d26-h8phb" Oct 31 19:52:22.257385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:22.257357 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-785d26-h8phb\" (UID: \"6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-785d26-h8phb" Oct 31 19:52:22.264350 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:22.264321 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-785d26-h8phb\" (UID: \"6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-785d26-h8phb" Oct 31 19:52:22.269195 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:22.269172 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvk5r\" (UniqueName: \"kubernetes.io/projected/6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52-kube-api-access-xvk5r\") pod \"maas-api-revoke-keys-e2e-inf-a-785d26-h8phb\" (UID: \"6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-785d26-h8phb" Oct 31 19:52:22.306427 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:22.306399 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-785d26-h8phb" Oct 31 19:52:22.666629 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:22.666593 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-785d26-h8phb"] Oct 31 19:52:22.667697 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:52:22.667671 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aaf2c61_c05e_4a25_9ffa_30f8bfa98b52.slice/crio-66fa18e25e68f645d549b9c41120fb8c489158e4b1a03f643d12199ec3a558a9 WatchSource:0}: Error finding container 66fa18e25e68f645d549b9c41120fb8c489158e4b1a03f643d12199ec3a558a9: Status 404 returned error can't find the container with id 66fa18e25e68f645d549b9c41120fb8c489158e4b1a03f643d12199ec3a558a9 Oct 31 19:52:23.276447 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:23.276403 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-785d26-h8phb" event={"ID":"6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52","Type":"ContainerStarted","Data":"2b36c53d49b3b90e6026df5e4950cff7ee7f0915974cf83215ae7e7570a58123"} Oct 31 19:52:23.276708 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:23.276686 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-785d26-h8phb" event={"ID":"6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52","Type":"ContainerStarted","Data":"66fa18e25e68f645d549b9c41120fb8c489158e4b1a03f643d12199ec3a558a9"} Oct 31 19:52:23.325632 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:23.325559 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-785d26-h8phb" podStartSLOduration=2.325535666 podStartE2EDuration="2.325535666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:23.303014716 +0000 UTC m=+1977.665835089" watchObservedRunningTime="2026-10-31 19:52:23.325535666 +0000 UTC m=+1977.688356040" Oct 31 19:52:24.291455 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:24.291337 4910 generic.go:356] "Generic (PLEG): container finished" podID="6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52" containerID="2b36c53d49b3b90e6026df5e4950cff7ee7f0915974cf83215ae7e7570a58123" exitCode=0 Oct 31 19:52:24.291847 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:24.291469 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-785d26-h8phb" event={"ID":"6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52","Type":"ContainerDied","Data":"2b36c53d49b3b90e6026df5e4950cff7ee7f0915974cf83215ae7e7570a58123"} Oct 31 19:52:25.749937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:25.745109 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-785d26-h8phb" Oct 31 19:52:25.853935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:25.853775 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52-kube-api-access-xvk5r\" (UniqueName: \"kubernetes.io/projected/6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52-kube-api-access-xvk5r\") pod \"6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52\" (UID: \"6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52\") " Oct 31 19:52:25.854138 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:25.853904 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52-maas-api-openshift-ca\") pod \"6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52\" (UID: \"6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52\") " Oct 31 19:52:25.858020 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:25.854207 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52-maas-api-xks-ca\") pod \"6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52\" (UID: \"6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52\") " Oct 31 19:52:25.858020 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:25.855275 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52-maas-api-openshift-ca" pod "6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52" (UID: "6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:52:25.866075 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:25.862087 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52-kube-api-access-xvk5r" pod "6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52" (UID: "6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52"). InnerVolumeSpecName "kube-api-access-xvk5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:52:25.948876 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:25.948758 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52-maas-api-xks-ca" pod "6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52" (UID: "6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:52:26.000055 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:25.957403 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:26.000055 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:25.981569 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvk5r\" (UniqueName: \"kubernetes.io/projected/6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52-kube-api-access-xvk5r\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:26.090937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.090240 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:26.313388 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.313356 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-785d26-h8phb" Oct 31 19:52:26.313388 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.313381 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-785d26-h8phb" event={"ID":"6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52","Type":"ContainerDied","Data":"66fa18e25e68f645d549b9c41120fb8c489158e4b1a03f643d12199ec3a558a9"} Oct 31 19:52:26.313609 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.313432 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66fa18e25e68f645d549b9c41120fb8c489158e4b1a03f643d12199ec3a558a9" Oct 31 19:52:26.528171 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.528137 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7"] Oct 31 19:52:26.529030 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.528886 4910 cpu_manager.go:414] "containerMap: removing container" podUID="6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52" containerName="revoke-keys" Oct 31 19:52:26.529030 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.528924 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52" containerName="revoke-keys" Oct 31 19:52:26.529215 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.529066 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52" containerName="revoke-keys" Oct 31 19:52:26.532822 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.532801 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.572351 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.569761 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-d9c048-openshift-default-dockercfg-fk2wl\"" Oct 31 19:52:26.573514 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.573488 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7"] Oct 31 19:52:26.598996 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.598958 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-credential-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.599173 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.599013 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2ccc\" (UniqueName: \"kubernetes.io/projected/64efa0c3-abc5-4253-8f01-cb9954e33dc6-kube-api-access-h2ccc\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.599173 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.599071 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-workload-certs\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.599173 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.599122 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-token\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.599353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.599201 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-envoy\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.599353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.599264 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-podinfo\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.599353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.599309 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-data\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.599514 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.599355 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istiod-ca-cert\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.599573 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.599526 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-workload-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.768989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.704876 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-token\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.768989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.705001 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-envoy\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.768989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.705072 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-podinfo\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.768989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.705132 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-data\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.768989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.705182 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istiod-ca-cert\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.768989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.705276 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-workload-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.768989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.705433 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-credential-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.768989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.705465 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2ccc\" (UniqueName: \"kubernetes.io/projected/64efa0c3-abc5-4253-8f01-cb9954e33dc6-kube-api-access-h2ccc\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.768989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.705519 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-workload-certs\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.768989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.706034 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-workload-certs\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.768989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.707565 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istiod-ca-cert\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.768989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.741325 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-data\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.768989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.741591 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-credential-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.768989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.768921 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-workload-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.913021 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.912623 4910 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 31 19:52:26.913021 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.912682 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" podUID="25d970e7-85b8-4057-8b20-5df84679b3ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 31 19:52:26.950015 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.948569 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-token\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.958239 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.951926 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-podinfo\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.962146 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.962082 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-envoy\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:26.965812 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:26.965760 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2ccc\" (UniqueName: \"kubernetes.io/projected/64efa0c3-abc5-4253-8f01-cb9954e33dc6-kube-api-access-h2ccc\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:27.120203 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.120171 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:52:27.196592 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.196500 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:27.241243 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.241206 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-podinfo\") pod \"25d970e7-85b8-4057-8b20-5df84679b3ab\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " Oct 31 19:52:27.241432 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.241353 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-token\" (UniqueName: \"kubernetes.io/projected/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-token\") pod \"25d970e7-85b8-4057-8b20-5df84679b3ab\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " Oct 31 19:52:27.241432 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.241415 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-data\") pod \"25d970e7-85b8-4057-8b20-5df84679b3ab\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " Oct 31 19:52:27.241562 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.241480 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-workload-certs\") pod \"25d970e7-85b8-4057-8b20-5df84679b3ab\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " Oct 31 19:52:27.241562 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.241534 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-workload-socket\") pod \"25d970e7-85b8-4057-8b20-5df84679b3ab\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " Oct 31 19:52:27.241667 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.241585 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d970e7-85b8-4057-8b20-5df84679b3ab-kube-api-access-kf6vw\" (UniqueName: \"kubernetes.io/projected/25d970e7-85b8-4057-8b20-5df84679b3ab-kube-api-access-kf6vw\") pod \"25d970e7-85b8-4057-8b20-5df84679b3ab\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " Oct 31 19:52:27.241667 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.241657 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25d970e7-85b8-4057-8b20-5df84679b3ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d970e7-85b8-4057-8b20-5df84679b3ab-istiod-ca-cert\") pod \"25d970e7-85b8-4057-8b20-5df84679b3ab\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " Oct 31 19:52:27.241781 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.241687 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-data" pod "25d970e7-85b8-4057-8b20-5df84679b3ab" (UID: "25d970e7-85b8-4057-8b20-5df84679b3ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:52:27.241781 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.241703 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-workload-certs" pod "25d970e7-85b8-4057-8b20-5df84679b3ab" (UID: "25d970e7-85b8-4057-8b20-5df84679b3ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:52:27.241781 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.241747 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-envoy\") pod \"25d970e7-85b8-4057-8b20-5df84679b3ab\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " Oct 31 19:52:27.241962 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.241814 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-workload-socket" pod "25d970e7-85b8-4057-8b20-5df84679b3ab" (UID: "25d970e7-85b8-4057-8b20-5df84679b3ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:52:27.241962 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.241878 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-credential-socket\") pod \"25d970e7-85b8-4057-8b20-5df84679b3ab\" (UID: \"25d970e7-85b8-4057-8b20-5df84679b3ab\") " Oct 31 19:52:27.242082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.242037 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d970e7-85b8-4057-8b20-5df84679b3ab-istiod-ca-cert" pod "25d970e7-85b8-4057-8b20-5df84679b3ab" (UID: "25d970e7-85b8-4057-8b20-5df84679b3ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:52:27.242316 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.242292 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-credential-socket" pod "25d970e7-85b8-4057-8b20-5df84679b3ab" (UID: "25d970e7-85b8-4057-8b20-5df84679b3ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:52:27.242643 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.242614 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:27.242740 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.242643 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:27.242740 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.242660 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:27.242740 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.242674 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:27.242740 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.242689 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d970e7-85b8-4057-8b20-5df84679b3ab-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:27.244114 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.244085 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d970e7-85b8-4057-8b20-5df84679b3ab-kube-api-access-kf6vw" pod "25d970e7-85b8-4057-8b20-5df84679b3ab" (UID: "25d970e7-85b8-4057-8b20-5df84679b3ab"). InnerVolumeSpecName "kube-api-access-kf6vw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:52:27.244114 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.244097 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-token" pod "25d970e7-85b8-4057-8b20-5df84679b3ab" (UID: "25d970e7-85b8-4057-8b20-5df84679b3ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:52:27.244512 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.244484 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-podinfo" pod "25d970e7-85b8-4057-8b20-5df84679b3ab" (UID: "25d970e7-85b8-4057-8b20-5df84679b3ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:52:27.245364 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.245345 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-envoy" pod "25d970e7-85b8-4057-8b20-5df84679b3ab" (UID: "25d970e7-85b8-4057-8b20-5df84679b3ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:52:27.363472 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.348846 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:27.363472 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.348885 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:27.372999 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.348902 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf6vw\" (UniqueName: \"kubernetes.io/projected/25d970e7-85b8-4057-8b20-5df84679b3ab-kube-api-access-kf6vw\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:27.372999 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.372957 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d970e7-85b8-4057-8b20-5df84679b3ab-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:27.394929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.389635 4910 generic.go:356] "Generic (PLEG): container finished" podID="25d970e7-85b8-4057-8b20-5df84679b3ab" containerID="81d1f5dc97000d0eeb0a8510c17f6c12563b497d0dafe57db65ddacc09daa26d" exitCode=0 Oct 31 19:52:27.394929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.389703 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" event={"ID":"25d970e7-85b8-4057-8b20-5df84679b3ab","Type":"ContainerDied","Data":"81d1f5dc97000d0eeb0a8510c17f6c12563b497d0dafe57db65ddacc09daa26d"} Oct 31 19:52:27.394929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.389751 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" event={"ID":"25d970e7-85b8-4057-8b20-5df84679b3ab","Type":"ContainerDied","Data":"41fab0b24e083d02b5ac83458e968ee9bffa9f6ff3ee31504d457f098f9b67d9"} Oct 31 19:52:27.394929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.389779 4910 scope.go:122] "RemoveContainer" containerID="81d1f5dc97000d0eeb0a8510c17f6c12563b497d0dafe57db65ddacc09daa26d" Oct 31 19:52:27.394929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.389957 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t" Oct 31 19:52:27.415856 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.415829 4910 scope.go:122] "RemoveContainer" containerID="81d1f5dc97000d0eeb0a8510c17f6c12563b497d0dafe57db65ddacc09daa26d" Oct 31 19:52:27.416387 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:52:27.416366 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81d1f5dc97000d0eeb0a8510c17f6c12563b497d0dafe57db65ddacc09daa26d\": container with ID starting with 81d1f5dc97000d0eeb0a8510c17f6c12563b497d0dafe57db65ddacc09daa26d not found: ID does not exist" containerID="81d1f5dc97000d0eeb0a8510c17f6c12563b497d0dafe57db65ddacc09daa26d" Oct 31 19:52:27.416531 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.416507 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d1f5dc97000d0eeb0a8510c17f6c12563b497d0dafe57db65ddacc09daa26d"} err="failed to get container status \"81d1f5dc97000d0eeb0a8510c17f6c12563b497d0dafe57db65ddacc09daa26d\": rpc error: code = NotFound desc = could not find container \"81d1f5dc97000d0eeb0a8510c17f6c12563b497d0dafe57db65ddacc09daa26d\": container with ID starting with 81d1f5dc97000d0eeb0a8510c17f6c12563b497d0dafe57db65ddacc09daa26d not found: ID does not exist" Oct 31 19:52:27.458742 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.458662 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t"] Oct 31 19:52:27.476027 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.475989 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-g4z7t"] Oct 31 19:52:27.508006 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:52:27.507970 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64efa0c3_abc5_4253_8f01_cb9954e33dc6.slice/crio-5207a1e72f6aedc629ffa66ffb3a5ec89ebf0c9cf983324cebc6f0b086463712 WatchSource:0}: Error finding container 5207a1e72f6aedc629ffa66ffb3a5ec89ebf0c9cf983324cebc6f0b086463712: Status 404 returned error can't find the container with id 5207a1e72f6aedc629ffa66ffb3a5ec89ebf0c9cf983324cebc6f0b086463712 Oct 31 19:52:27.508824 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.508798 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7"] Oct 31 19:52:27.521970 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.512276 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:52:27.521970 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.512449 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:52:27.521970 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.512552 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:52:27.761676 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:27.761578 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25d970e7-85b8-4057-8b20-5df84679b3ab" path="/var/lib/kubelet/pods/25d970e7-85b8-4057-8b20-5df84679b3ab/volumes" Oct 31 19:52:28.405493 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:28.405444 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" event={"ID":"64efa0c3-abc5-4253-8f01-cb9954e33dc6","Type":"ContainerStarted","Data":"f916b44d93c7fa872d1ae3cf94864bdcb7690d0977fabdc11b5e37bad0588312"} Oct 31 19:52:28.406008 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:28.405517 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" event={"ID":"64efa0c3-abc5-4253-8f01-cb9954e33dc6","Type":"ContainerStarted","Data":"5207a1e72f6aedc629ffa66ffb3a5ec89ebf0c9cf983324cebc6f0b086463712"} Oct 31 19:52:28.469907 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:28.469832 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" podStartSLOduration=2.469811231 podStartE2EDuration="2.469811231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:28.462498524 +0000 UTC m=+1982.825318898" watchObservedRunningTime="2026-10-31 19:52:28.469811231 +0000 UTC m=+1982.832631604" Oct 31 19:52:29.197603 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:29.197563 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:29.204761 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:29.204733 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:29.419453 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:29.419331 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:29.421051 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:29.420900 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:52:36.499262 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.499224 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t"] Oct 31 19:52:36.499893 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.499876 4910 cpu_manager.go:414] "containerMap: removing container" podUID="25d970e7-85b8-4057-8b20-5df84679b3ab" containerName="istio-proxy" Oct 31 19:52:36.499995 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.499893 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d970e7-85b8-4057-8b20-5df84679b3ab" containerName="istio-proxy" Oct 31 19:52:36.500054 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.500019 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d970e7-85b8-4057-8b20-5df84679b3ab" containerName="istio-proxy" Oct 31 19:52:36.505277 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.505255 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.522013 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.521984 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-e618e1-openshift-default-dockercfg-h496q\"" Oct 31 19:52:36.530099 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.530068 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t"] Oct 31 19:52:36.592826 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.592768 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnlkn\" (UniqueName: \"kubernetes.io/projected/10538611-7fbc-4a9c-b7f9-61c479d57fc8-kube-api-access-bnlkn\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.593097 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.592919 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-workload-certs\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.593097 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.592992 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-data\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.593244 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.593150 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-podinfo\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.593244 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.593196 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-envoy\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.593355 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.593291 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-credential-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.593355 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.593330 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-workload-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.593478 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.593379 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istiod-ca-cert\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.593478 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.593417 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-token\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.699932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.696504 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnlkn\" (UniqueName: \"kubernetes.io/projected/10538611-7fbc-4a9c-b7f9-61c479d57fc8-kube-api-access-bnlkn\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.699932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.696693 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-workload-certs\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.699932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.696797 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-data\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.699932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.696879 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-podinfo\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.699932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.696924 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-envoy\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.699932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.696998 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-credential-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.699932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.697057 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-workload-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.699932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.697096 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istiod-ca-cert\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.699932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.697150 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-token\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.702967 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.702316 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-workload-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.703219 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.703190 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istiod-ca-cert\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.703481 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.703457 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-workload-certs\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.704021 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.703990 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-data\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.704479 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.704443 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-credential-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.712368 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.712239 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-envoy\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.713673 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.713601 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-podinfo\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.749061 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.748983 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnlkn\" (UniqueName: \"kubernetes.io/projected/10538611-7fbc-4a9c-b7f9-61c479d57fc8-kube-api-access-bnlkn\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.772028 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.771805 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-token\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:36.823836 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:36.823674 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:37.119844 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:52:37.119811 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10538611_7fbc_4a9c_b7f9_61c479d57fc8.slice/crio-7e234628d0f7f7bbb31fc14a8d2934d5f3984a9646b8334e5ac4a1ab65fe3b27 WatchSource:0}: Error finding container 7e234628d0f7f7bbb31fc14a8d2934d5f3984a9646b8334e5ac4a1ab65fe3b27: Status 404 returned error can't find the container with id 7e234628d0f7f7bbb31fc14a8d2934d5f3984a9646b8334e5ac4a1ab65fe3b27 Oct 31 19:52:37.120731 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:37.120591 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t"] Oct 31 19:52:37.123595 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:37.123549 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:52:37.123705 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:37.123642 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:52:37.123705 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:37.123685 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:52:37.670357 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:37.670307 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" event={"ID":"10538611-7fbc-4a9c-b7f9-61c479d57fc8","Type":"ContainerStarted","Data":"d37cbe366f1bf8e80c2ca3f859432b1d6ed6cd443573dd1d4f6c12de46ae6724"} Oct 31 19:52:37.670864 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:37.670375 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" event={"ID":"10538611-7fbc-4a9c-b7f9-61c479d57fc8","Type":"ContainerStarted","Data":"7e234628d0f7f7bbb31fc14a8d2934d5f3984a9646b8334e5ac4a1ab65fe3b27"} Oct 31 19:52:37.708849 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:37.708794 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" podStartSLOduration=1.708778976 podStartE2EDuration="1.708778976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:37.701718402 +0000 UTC m=+1992.064538775" watchObservedRunningTime="2026-10-31 19:52:37.708778976 +0000 UTC m=+1992.071599347" Oct 31 19:52:37.824324 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:37.824282 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:38.843090 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:38.843049 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 19:52:38.843705 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:38.843666 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" podUID="10538611-7fbc-4a9c-b7f9-61c479d57fc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 19:52:39.842934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:39.839607 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 19:52:39.842934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:39.839724 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" podUID="10538611-7fbc-4a9c-b7f9-61c479d57fc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 19:52:40.833359 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:40.833317 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 19:52:40.833654 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:40.833626 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" podUID="10538611-7fbc-4a9c-b7f9-61c479d57fc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 19:52:41.824328 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:41.824289 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 19:52:41.824932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:41.824882 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" podUID="10538611-7fbc-4a9c-b7f9-61c479d57fc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 19:52:42.721937 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:52:42.720367 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:52:42.828935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:42.828299 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 31 19:52:42.828935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:42.828365 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" podUID="10538611-7fbc-4a9c-b7f9-61c479d57fc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 31 19:52:44.002264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:44.002227 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:44.009499 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:44.009469 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:44.018188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:44.018161 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 19:52:49.946465 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:49.946422 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj"] Oct 31 19:52:49.947300 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:49.947266 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj" podUID="d59df105-1b29-4a08-8677-69af26994de8" containerName="maas-api" containerID="cri-o://a72ce22618558c704de7125c3b7b380f86c9620e0d3187722f9a10be1c97da7f" gracePeriod=30 Oct 31 19:52:50.472936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:50.453551 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-785d26-h8phb"] Oct 31 19:52:50.507931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:50.506482 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-785d26-h8phb"] Oct 31 19:52:50.647407 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:50.647376 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj" Oct 31 19:52:50.760523 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:50.760423 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d59df105-1b29-4a08-8677-69af26994de8-kube-api-access-m5r5h\" (UniqueName: \"kubernetes.io/projected/d59df105-1b29-4a08-8677-69af26994de8-kube-api-access-m5r5h\") pod \"d59df105-1b29-4a08-8677-69af26994de8\" (UID: \"d59df105-1b29-4a08-8677-69af26994de8\") " Oct 31 19:52:50.760709 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:50.760582 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d59df105-1b29-4a08-8677-69af26994de8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59df105-1b29-4a08-8677-69af26994de8-metrics-certs\") pod \"d59df105-1b29-4a08-8677-69af26994de8\" (UID: \"d59df105-1b29-4a08-8677-69af26994de8\") " Oct 31 19:52:50.760709 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:50.760630 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d59df105-1b29-4a08-8677-69af26994de8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d59df105-1b29-4a08-8677-69af26994de8-maas-api-tls\") pod \"d59df105-1b29-4a08-8677-69af26994de8\" (UID: \"d59df105-1b29-4a08-8677-69af26994de8\") " Oct 31 19:52:50.760709 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:50.760681 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d59df105-1b29-4a08-8677-69af26994de8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59df105-1b29-4a08-8677-69af26994de8-openshift-service-ca\") pod \"d59df105-1b29-4a08-8677-69af26994de8\" (UID: \"d59df105-1b29-4a08-8677-69af26994de8\") " Oct 31 19:52:50.761613 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:50.761583 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d59df105-1b29-4a08-8677-69af26994de8-openshift-service-ca" pod "d59df105-1b29-4a08-8677-69af26994de8" (UID: "d59df105-1b29-4a08-8677-69af26994de8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:52:50.764589 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:50.764550 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d59df105-1b29-4a08-8677-69af26994de8-metrics-certs" pod "d59df105-1b29-4a08-8677-69af26994de8" (UID: "d59df105-1b29-4a08-8677-69af26994de8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:52:50.764589 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:50.764575 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59df105-1b29-4a08-8677-69af26994de8-kube-api-access-m5r5h" pod "d59df105-1b29-4a08-8677-69af26994de8" (UID: "d59df105-1b29-4a08-8677-69af26994de8"). InnerVolumeSpecName "kube-api-access-m5r5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:52:50.765274 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:50.765239 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d59df105-1b29-4a08-8677-69af26994de8-maas-api-tls" pod "d59df105-1b29-4a08-8677-69af26994de8" (UID: "d59df105-1b29-4a08-8677-69af26994de8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:52:50.810601 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:50.810563 4910 generic.go:356] "Generic (PLEG): container finished" podID="d59df105-1b29-4a08-8677-69af26994de8" containerID="a72ce22618558c704de7125c3b7b380f86c9620e0d3187722f9a10be1c97da7f" exitCode=0 Oct 31 19:52:50.810805 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:50.810608 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj" event={"ID":"d59df105-1b29-4a08-8677-69af26994de8","Type":"ContainerDied","Data":"a72ce22618558c704de7125c3b7b380f86c9620e0d3187722f9a10be1c97da7f"} Oct 31 19:52:50.810805 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:50.810623 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj" Oct 31 19:52:50.810805 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:50.810646 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj" event={"ID":"d59df105-1b29-4a08-8677-69af26994de8","Type":"ContainerDied","Data":"9b0918b954cd1267e11f1db6b59b6d1db93d53db26318dcdbc21db2e342c6e74"} Oct 31 19:52:50.810805 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:50.810667 4910 scope.go:122] "RemoveContainer" containerID="a72ce22618558c704de7125c3b7b380f86c9620e0d3187722f9a10be1c97da7f" Oct 31 19:52:50.831779 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:50.831750 4910 scope.go:122] "RemoveContainer" containerID="a72ce22618558c704de7125c3b7b380f86c9620e0d3187722f9a10be1c97da7f" Oct 31 19:52:50.832299 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:52:50.832266 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a72ce22618558c704de7125c3b7b380f86c9620e0d3187722f9a10be1c97da7f\": container with ID starting with a72ce22618558c704de7125c3b7b380f86c9620e0d3187722f9a10be1c97da7f not found: ID does not exist" containerID="a72ce22618558c704de7125c3b7b380f86c9620e0d3187722f9a10be1c97da7f" Oct 31 19:52:50.832453 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:50.832311 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a72ce22618558c704de7125c3b7b380f86c9620e0d3187722f9a10be1c97da7f"} err="failed to get container status \"a72ce22618558c704de7125c3b7b380f86c9620e0d3187722f9a10be1c97da7f\": rpc error: code = NotFound desc = could not find container \"a72ce22618558c704de7125c3b7b380f86c9620e0d3187722f9a10be1c97da7f\": container with ID starting with a72ce22618558c704de7125c3b7b380f86c9620e0d3187722f9a10be1c97da7f not found: ID does not exist" Oct 31 19:52:50.845153 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:50.845126 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj"] Oct 31 19:52:50.849435 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:50.849409 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-785d26-7987f8dff4-h2vrj"] Oct 31 19:52:50.863578 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:50.863553 4910 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59df105-1b29-4a08-8677-69af26994de8-metrics-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:50.863578 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:50.863579 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d59df105-1b29-4a08-8677-69af26994de8-maas-api-tls\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:50.863751 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:50.863595 4910 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d59df105-1b29-4a08-8677-69af26994de8-openshift-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:50.863751 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:50.863608 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5r5h\" (UniqueName: \"kubernetes.io/projected/d59df105-1b29-4a08-8677-69af26994de8-kube-api-access-m5r5h\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:52:51.412940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.404094 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp"] Oct 31 19:52:51.412940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.404818 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d59df105-1b29-4a08-8677-69af26994de8" containerName="maas-api" Oct 31 19:52:51.412940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.404834 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d59df105-1b29-4a08-8677-69af26994de8" containerName="maas-api" Oct 31 19:52:51.412940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.405028 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d59df105-1b29-4a08-8677-69af26994de8" containerName="maas-api" Oct 31 19:52:51.412940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.410633 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp" Oct 31 19:52:51.420497 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.420468 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp"] Oct 31 19:52:51.421144 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.421100 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-d9c048-serving-cert\"" Oct 31 19:52:51.467105 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.467073 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4"] Oct 31 19:52:51.472373 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.472347 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.492433 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.491302 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-f5c1ab-openshift-default-dockercfg-r9nbk\"" Oct 31 19:52:51.492626 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.492570 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4"] Oct 31 19:52:51.574623 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.574592 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-podinfo\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.574792 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.574653 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-envoy\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.574792 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.574725 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-token\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.574792 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.574754 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mlqq\" (UniqueName: \"kubernetes.io/projected/16950395-c37f-4c55-bc84-6be3d07c91f6-kube-api-access-4mlqq\") pod \"maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp\" (UID: \"16950395-c37f-4c55-bc84-6be3d07c91f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp" Oct 31 19:52:51.574920 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.574843 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-credential-socket\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.574920 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.574877 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8398426-b36a-4c2f-b74a-b6676b915a5c-istiod-ca-cert\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.575007 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.574971 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-workload-certs\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.575041 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.575029 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt8x9\" (UniqueName: \"kubernetes.io/projected/b8398426-b36a-4c2f-b74a-b6676b915a5c-kube-api-access-xt8x9\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.575192 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.575165 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16950395-c37f-4c55-bc84-6be3d07c91f6-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp\" (UID: \"16950395-c37f-4c55-bc84-6be3d07c91f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp" Oct 31 19:52:51.575294 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.575246 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-workload-socket\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.575365 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.575298 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16950395-c37f-4c55-bc84-6be3d07c91f6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp\" (UID: \"16950395-c37f-4c55-bc84-6be3d07c91f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp" Oct 31 19:52:51.575462 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.575441 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-data\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.575583 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.575557 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16950395-c37f-4c55-bc84-6be3d07c91f6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp\" (UID: \"16950395-c37f-4c55-bc84-6be3d07c91f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp" Oct 31 19:52:51.677967 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.677850 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt8x9\" (UniqueName: \"kubernetes.io/projected/b8398426-b36a-4c2f-b74a-b6676b915a5c-kube-api-access-xt8x9\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.678137 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.677974 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16950395-c37f-4c55-bc84-6be3d07c91f6-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp\" (UID: \"16950395-c37f-4c55-bc84-6be3d07c91f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp" Oct 31 19:52:51.678203 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.678133 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-workload-socket\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.678203 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.678190 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16950395-c37f-4c55-bc84-6be3d07c91f6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp\" (UID: \"16950395-c37f-4c55-bc84-6be3d07c91f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp" Oct 31 19:52:51.678321 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.678251 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-data\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.678321 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.678275 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16950395-c37f-4c55-bc84-6be3d07c91f6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp\" (UID: \"16950395-c37f-4c55-bc84-6be3d07c91f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp" Oct 31 19:52:51.678437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.678365 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-podinfo\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.678437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.678407 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-envoy\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.678437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.678433 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-token\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.678599 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.678448 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mlqq\" (UniqueName: \"kubernetes.io/projected/16950395-c37f-4c55-bc84-6be3d07c91f6-kube-api-access-4mlqq\") pod \"maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp\" (UID: \"16950395-c37f-4c55-bc84-6be3d07c91f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp" Oct 31 19:52:51.678599 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.678499 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-credential-socket\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.678599 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.678531 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8398426-b36a-4c2f-b74a-b6676b915a5c-istiod-ca-cert\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.678773 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.678600 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-workload-certs\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.678773 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.678746 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-data\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.678995 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.678973 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-workload-socket\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.679157 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.679022 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-workload-certs\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.679537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.679507 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-credential-socket\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.679963 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.679892 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8398426-b36a-4c2f-b74a-b6676b915a5c-istiod-ca-cert\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.680268 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.680247 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16950395-c37f-4c55-bc84-6be3d07c91f6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp\" (UID: \"16950395-c37f-4c55-bc84-6be3d07c91f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp" Oct 31 19:52:51.681603 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.681506 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16950395-c37f-4c55-bc84-6be3d07c91f6-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp\" (UID: \"16950395-c37f-4c55-bc84-6be3d07c91f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp" Oct 31 19:52:51.682342 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.682315 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16950395-c37f-4c55-bc84-6be3d07c91f6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp\" (UID: \"16950395-c37f-4c55-bc84-6be3d07c91f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp" Oct 31 19:52:51.682433 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.682365 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-envoy\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.683404 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.683382 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-podinfo\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.693126 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.693099 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt8x9\" (UniqueName: \"kubernetes.io/projected/b8398426-b36a-4c2f-b74a-b6676b915a5c-kube-api-access-xt8x9\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.693261 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.693173 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-token\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:51.693754 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.693733 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mlqq\" (UniqueName: \"kubernetes.io/projected/16950395-c37f-4c55-bc84-6be3d07c91f6-kube-api-access-4mlqq\") pod \"maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp\" (UID: \"16950395-c37f-4c55-bc84-6be3d07c91f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp" Oct 31 19:52:51.735693 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.735661 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp" Oct 31 19:52:51.759194 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.759163 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52" path="/var/lib/kubelet/pods/6aaf2c61-c05e-4a25-9ffa-30f8bfa98b52/volumes" Oct 31 19:52:51.760137 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.760116 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d59df105-1b29-4a08-8677-69af26994de8" path="/var/lib/kubelet/pods/d59df105-1b29-4a08-8677-69af26994de8/volumes" Oct 31 19:52:51.788967 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:51.788584 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:52.062079 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.062049 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp"] Oct 31 19:52:52.063046 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:52:52.063014 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16950395_c37f_4c55_bc84_6be3d07c91f6.slice/crio-8f38ec94b53c3ecfb600237a996da4602dda83bcd8fc7366d3eac2c77c7729d0 WatchSource:0}: Error finding container 8f38ec94b53c3ecfb600237a996da4602dda83bcd8fc7366d3eac2c77c7729d0: Status 404 returned error can't find the container with id 8f38ec94b53c3ecfb600237a996da4602dda83bcd8fc7366d3eac2c77c7729d0 Oct 31 19:52:52.064888 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.064655 4910 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 19:52:52.262936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.238393 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4"] Oct 31 19:52:52.262936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.237306 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:52:52.262936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.238961 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:52:52.262936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.239023 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:52:52.459992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.446867 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk"] Oct 31 19:52:52.469126 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.465963 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.469126 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.468955 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk"] Oct 31 19:52:52.633935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.633438 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-credential-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.633935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.633612 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-workload-certs\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.651609 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.651562 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-envoy\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.651960 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.651932 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-workload-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.652113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.652097 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-data\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.652271 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.652247 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istiod-ca-cert\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.652473 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.652457 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-token\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.652677 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.652659 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2j4f\" (UniqueName: \"kubernetes.io/projected/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-kube-api-access-j2j4f\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.652809 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.652794 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-podinfo\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.754791 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.754757 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-workload-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.754960 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.754806 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-data\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.754960 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.754849 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istiod-ca-cert\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.754960 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.754928 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-token\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.755159 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.754990 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2j4f\" (UniqueName: \"kubernetes.io/projected/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-kube-api-access-j2j4f\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.755159 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.755018 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-podinfo\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.755159 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.755141 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-credential-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.755327 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.755198 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-workload-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.755327 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.755282 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-workload-certs\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.755431 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.755367 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-envoy\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.755524 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.755500 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-credential-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.755794 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.755771 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-data\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.757216 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.757176 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-workload-certs\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.758472 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.758430 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istiod-ca-cert\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.763580 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.763554 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-envoy\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.774058 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.769424 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-podinfo\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.774058 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.772457 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-token\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.774058 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.773365 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2j4f\" (UniqueName: \"kubernetes.io/projected/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-kube-api-access-j2j4f\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.794625 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.794538 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:52.837339 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.835783 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" event={"ID":"b8398426-b36a-4c2f-b74a-b6676b915a5c","Type":"ContainerStarted","Data":"5f65a5603972a32ad5d1d1b4af8986e1739153182342d54315a54a49e955cbc4"} Oct 31 19:52:52.837339 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.835856 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" event={"ID":"b8398426-b36a-4c2f-b74a-b6676b915a5c","Type":"ContainerStarted","Data":"859f6411469f4e47437df05a36db04665149f59e9b383b158faaf932438b8b0e"} Oct 31 19:52:52.839435 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.839407 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp" event={"ID":"16950395-c37f-4c55-bc84-6be3d07c91f6","Type":"ContainerStarted","Data":"cb5202b2ec1528bc21f48fdd1a67957a6403d3d6dd02f468eeff069f37432934"} Oct 31 19:52:52.839631 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.839612 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp" event={"ID":"16950395-c37f-4c55-bc84-6be3d07c91f6","Type":"ContainerStarted","Data":"8f38ec94b53c3ecfb600237a996da4602dda83bcd8fc7366d3eac2c77c7729d0"} Oct 31 19:52:52.840566 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.840547 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp" Oct 31 19:52:52.878282 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.875066 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" podStartSLOduration=1.8750269419999999 podStartE2EDuration="1.875026942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:52.867340385 +0000 UTC m=+2007.230160769" watchObservedRunningTime="2026-10-31 19:52:52.875026942 +0000 UTC m=+2007.237847315" Oct 31 19:52:52.908721 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:52.908664 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp" podStartSLOduration=1.311774182 podStartE2EDuration="1.908644149s" totalImagesPullingTime="596.869967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:52.896512907 +0000 UTC m=+2007.259333281" watchObservedRunningTime="2026-10-31 19:52:52.908644149 +0000 UTC m=+2007.271464522" Oct 31 19:52:53.130125 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:53.130082 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk"] Oct 31 19:52:53.134215 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:53.134170 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:52:53.136865 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:53.136830 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:52:53.137136 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:53.137109 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:52:53.790931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:53.790122 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:53.791348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:53.791051 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 31 19:52:53.791348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:53.791131 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" podUID="b8398426-b36a-4c2f-b74a-b6676b915a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 31 19:52:53.935358 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:53.934675 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" event={"ID":"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0","Type":"ContainerStarted","Data":"aba5df7861a9f680491ef00e55f4b40eade180552490970cc19cea0a2cb3733b"} Oct 31 19:52:53.935358 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:53.934737 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" event={"ID":"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0","Type":"ContainerStarted","Data":"e76df673bb84f2da56360138edafd407f7015b069ce80d01a14e54d8a5ff193f"} Oct 31 19:52:54.004554 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:54.004387 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" podStartSLOduration=2.004368277 podStartE2EDuration="2.004368277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:54.000045475 +0000 UTC m=+2008.362865868" watchObservedRunningTime="2026-10-31 19:52:54.004368277 +0000 UTC m=+2008.367188653" Oct 31 19:52:54.798066 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:54.798033 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:52:54.798700 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:54.798307 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 31 19:52:54.798895 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:54.798870 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" podUID="b8398426-b36a-4c2f-b74a-b6676b915a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 31 19:52:54.800537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:54.800518 4910 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 31 19:52:54.800695 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:54.800673 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" podUID="c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 31 19:52:55.790492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:55.790095 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 31 19:52:55.790492 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:55.790162 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" podUID="b8398426-b36a-4c2f-b74a-b6676b915a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 31 19:52:55.796068 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:55.796038 4910 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 31 19:52:55.796196 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:55.796087 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" podUID="c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 31 19:52:56.791647 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:56.791610 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 31 19:52:56.792132 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:56.791667 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" podUID="b8398426-b36a-4c2f-b74a-b6676b915a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 31 19:52:56.795201 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:56.795157 4910 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 31 19:52:56.795351 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:56.795207 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" podUID="c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 31 19:52:57.789933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:57.789346 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 31 19:52:57.789933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:57.789406 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" podUID="b8398426-b36a-4c2f-b74a-b6676b915a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 31 19:52:57.797421 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:57.795216 4910 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 31 19:52:57.797421 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:57.795277 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" podUID="c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 31 19:52:58.819932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:58.804661 4910 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 31 19:52:58.819932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:58.817655 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" podUID="c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 31 19:52:58.822027 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:58.804733 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 31 19:52:58.822291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:58.822248 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" podUID="b8398426-b36a-4c2f-b74a-b6676b915a5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 31 19:52:59.261277 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:59.003740 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb"] Oct 31 19:52:59.279197 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:59.279144 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" podUID="7631819a-de14-4a11-a5a9-9b2e05e7c7a6" containerName="istio-proxy" containerID="cri-o://f67913e4013c7eead267489dbcc8a1556ee654c25b9ebefa9d6bb63dd6592bd0" gracePeriod=30 Oct 31 19:52:59.405926 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:59.405874 4910 patch_prober.go:29] interesting pod/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:52:59.406115 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:59.405958 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" podUID="7631819a-de14-4a11-a5a9-9b2e05e7c7a6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:52:59.504169 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:59.503379 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8"] Oct 31 19:52:59.511439 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:59.511370 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8" Oct 31 19:52:59.565550 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:59.565519 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8"] Oct 31 19:52:59.586758 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:59.586709 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:52:59.586953 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:59.586835 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:52:59.602148 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:59.602059 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8814ab6-b8e3-44b3-a145-d16a6b128e49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8\" (UID: \"f8814ab6-b8e3-44b3-a145-d16a6b128e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8" Oct 31 19:52:59.602316 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:59.602238 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drftl\" (UniqueName: \"kubernetes.io/projected/f8814ab6-b8e3-44b3-a145-d16a6b128e49-kube-api-access-drftl\") pod \"maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8\" (UID: \"f8814ab6-b8e3-44b3-a145-d16a6b128e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8" Oct 31 19:52:59.602427 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:59.602389 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8814ab6-b8e3-44b3-a145-d16a6b128e49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8\" (UID: \"f8814ab6-b8e3-44b3-a145-d16a6b128e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8" Oct 31 19:52:59.713099 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:59.707094 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8814ab6-b8e3-44b3-a145-d16a6b128e49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8\" (UID: \"f8814ab6-b8e3-44b3-a145-d16a6b128e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8" Oct 31 19:52:59.713099 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:59.707231 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drftl\" (UniqueName: \"kubernetes.io/projected/f8814ab6-b8e3-44b3-a145-d16a6b128e49-kube-api-access-drftl\") pod \"maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8\" (UID: \"f8814ab6-b8e3-44b3-a145-d16a6b128e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8" Oct 31 19:52:59.713099 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:59.707305 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8814ab6-b8e3-44b3-a145-d16a6b128e49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8\" (UID: \"f8814ab6-b8e3-44b3-a145-d16a6b128e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8" Oct 31 19:52:59.713099 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:59.708602 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8814ab6-b8e3-44b3-a145-d16a6b128e49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8\" (UID: \"f8814ab6-b8e3-44b3-a145-d16a6b128e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8" Oct 31 19:52:59.713099 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:59.710838 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8814ab6-b8e3-44b3-a145-d16a6b128e49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8\" (UID: \"f8814ab6-b8e3-44b3-a145-d16a6b128e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8" Oct 31 19:52:59.724290 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:59.722252 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drftl\" (UniqueName: \"kubernetes.io/projected/f8814ab6-b8e3-44b3-a145-d16a6b128e49-kube-api-access-drftl\") pod \"maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8\" (UID: \"f8814ab6-b8e3-44b3-a145-d16a6b128e49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8" Oct 31 19:52:59.796837 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:59.796692 4910 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 31 19:52:59.796837 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:59.796793 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" podUID="c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 31 19:52:59.797170 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:59.797145 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:59.799250 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:59.798339 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:59.803410 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:59.803383 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 19:52:59.907804 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:59.907723 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8" Oct 31 19:52:59.980961 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:52:59.978716 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp" Oct 31 19:53:00.322093 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:00.322062 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8"] Oct 31 19:53:00.352320 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:53:00.352227 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8814ab6_b8e3_44b3_a145_d16a6b128e49.slice/crio-5356414a4ec53912d6b225d7e9de0ba9e69333462896c4d21c0604e448d37fa4 WatchSource:0}: Error finding container 5356414a4ec53912d6b225d7e9de0ba9e69333462896c4d21c0604e448d37fa4: Status 404 returned error can't find the container with id 5356414a4ec53912d6b225d7e9de0ba9e69333462896c4d21c0604e448d37fa4 Oct 31 19:53:00.802011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:00.801977 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:53:00.802298 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:00.802272 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:53:00.803220 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:00.803196 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:53:01.272898 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:01.272860 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8" event={"ID":"f8814ab6-b8e3-44b3-a145-d16a6b128e49","Type":"ContainerStarted","Data":"43219138eea25a83520457e9499fe98188516b3ab1bffceec0704b56ec17785b"} Oct 31 19:53:01.273341 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:01.272958 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8" event={"ID":"f8814ab6-b8e3-44b3-a145-d16a6b128e49","Type":"ContainerStarted","Data":"5356414a4ec53912d6b225d7e9de0ba9e69333462896c4d21c0604e448d37fa4"} Oct 31 19:53:01.297586 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:01.297516 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8" podStartSLOduration=2.297497388 podStartE2EDuration="2.297497388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:53:01.291747348 +0000 UTC m=+2015.654567722" watchObservedRunningTime="2026-10-31 19:53:01.297497388 +0000 UTC m=+2015.660317762" Oct 31 19:53:02.282886 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:02.282849 4910 generic.go:356] "Generic (PLEG): container finished" podID="f8814ab6-b8e3-44b3-a145-d16a6b128e49" containerID="43219138eea25a83520457e9499fe98188516b3ab1bffceec0704b56ec17785b" exitCode=0 Oct 31 19:53:02.283425 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:02.282935 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8" event={"ID":"f8814ab6-b8e3-44b3-a145-d16a6b128e49","Type":"ContainerDied","Data":"43219138eea25a83520457e9499fe98188516b3ab1bffceec0704b56ec17785b"} Oct 31 19:53:03.486061 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:03.486033 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8" Oct 31 19:53:03.535938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:03.535649 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8814ab6-b8e3-44b3-a145-d16a6b128e49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8814ab6-b8e3-44b3-a145-d16a6b128e49-maas-api-openshift-ca\") pod \"f8814ab6-b8e3-44b3-a145-d16a6b128e49\" (UID: \"f8814ab6-b8e3-44b3-a145-d16a6b128e49\") " Oct 31 19:53:03.535938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:03.535835 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8814ab6-b8e3-44b3-a145-d16a6b128e49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8814ab6-b8e3-44b3-a145-d16a6b128e49-maas-api-xks-ca\") pod \"f8814ab6-b8e3-44b3-a145-d16a6b128e49\" (UID: \"f8814ab6-b8e3-44b3-a145-d16a6b128e49\") " Oct 31 19:53:03.535938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:03.535883 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8814ab6-b8e3-44b3-a145-d16a6b128e49-kube-api-access-drftl\" (UniqueName: \"kubernetes.io/projected/f8814ab6-b8e3-44b3-a145-d16a6b128e49-kube-api-access-drftl\") pod \"f8814ab6-b8e3-44b3-a145-d16a6b128e49\" (UID: \"f8814ab6-b8e3-44b3-a145-d16a6b128e49\") " Oct 31 19:53:03.537256 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:03.536382 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8814ab6-b8e3-44b3-a145-d16a6b128e49-maas-api-openshift-ca" pod "f8814ab6-b8e3-44b3-a145-d16a6b128e49" (UID: "f8814ab6-b8e3-44b3-a145-d16a6b128e49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:53:03.537401 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:03.537386 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8814ab6-b8e3-44b3-a145-d16a6b128e49-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:03.538957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:03.538929 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8814ab6-b8e3-44b3-a145-d16a6b128e49-maas-api-xks-ca" pod "f8814ab6-b8e3-44b3-a145-d16a6b128e49" (UID: "f8814ab6-b8e3-44b3-a145-d16a6b128e49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:53:03.580937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:03.568079 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8814ab6-b8e3-44b3-a145-d16a6b128e49-kube-api-access-drftl" pod "f8814ab6-b8e3-44b3-a145-d16a6b128e49" (UID: "f8814ab6-b8e3-44b3-a145-d16a6b128e49"). InnerVolumeSpecName "kube-api-access-drftl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:53:03.639719 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:03.639685 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8814ab6-b8e3-44b3-a145-d16a6b128e49-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:03.640032 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:03.640008 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drftl\" (UniqueName: \"kubernetes.io/projected/f8814ab6-b8e3-44b3-a145-d16a6b128e49-kube-api-access-drftl\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:04.302225 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.302195 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8" Oct 31 19:53:04.302423 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.302232 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8" event={"ID":"f8814ab6-b8e3-44b3-a145-d16a6b128e49","Type":"ContainerDied","Data":"5356414a4ec53912d6b225d7e9de0ba9e69333462896c4d21c0604e448d37fa4"} Oct 31 19:53:04.302423 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.302292 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5356414a4ec53912d6b225d7e9de0ba9e69333462896c4d21c0604e448d37fa4" Oct 31 19:53:04.631621 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.631591 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:53:04.673682 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.673648 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-token\" (UniqueName: \"kubernetes.io/projected/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-token\") pod \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " Oct 31 19:53:04.673887 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.673715 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-workload-socket\") pod \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " Oct 31 19:53:04.673887 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.673757 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-podinfo\") pod \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " Oct 31 19:53:04.673887 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.673831 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-kube-api-access-zrtwm\" (UniqueName: \"kubernetes.io/projected/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-kube-api-access-zrtwm\") pod \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " Oct 31 19:53:04.674100 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.673890 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-data\") pod \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " Oct 31 19:53:04.674100 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.673950 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istiod-ca-cert\") pod \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " Oct 31 19:53:04.674100 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.674011 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-workload-certs\") pod \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " Oct 31 19:53:04.674100 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.674064 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-envoy\") pod \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " Oct 31 19:53:04.674323 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.674138 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-credential-socket\") pod \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\" (UID: \"7631819a-de14-4a11-a5a9-9b2e05e7c7a6\") " Oct 31 19:53:04.675179 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.675039 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-credential-socket" pod "7631819a-de14-4a11-a5a9-9b2e05e7c7a6" (UID: "7631819a-de14-4a11-a5a9-9b2e05e7c7a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:04.675179 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.675039 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-workload-socket" pod "7631819a-de14-4a11-a5a9-9b2e05e7c7a6" (UID: "7631819a-de14-4a11-a5a9-9b2e05e7c7a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:04.675179 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.675070 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-data" pod "7631819a-de14-4a11-a5a9-9b2e05e7c7a6" (UID: "7631819a-de14-4a11-a5a9-9b2e05e7c7a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:04.675179 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.675156 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-workload-certs" pod "7631819a-de14-4a11-a5a9-9b2e05e7c7a6" (UID: "7631819a-de14-4a11-a5a9-9b2e05e7c7a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:04.675495 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.675438 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istiod-ca-cert" pod "7631819a-de14-4a11-a5a9-9b2e05e7c7a6" (UID: "7631819a-de14-4a11-a5a9-9b2e05e7c7a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:53:04.700782 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.699886 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-envoy" pod "7631819a-de14-4a11-a5a9-9b2e05e7c7a6" (UID: "7631819a-de14-4a11-a5a9-9b2e05e7c7a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:04.700782 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.699895 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-podinfo" pod "7631819a-de14-4a11-a5a9-9b2e05e7c7a6" (UID: "7631819a-de14-4a11-a5a9-9b2e05e7c7a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:53:04.700782 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.699974 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-token" pod "7631819a-de14-4a11-a5a9-9b2e05e7c7a6" (UID: "7631819a-de14-4a11-a5a9-9b2e05e7c7a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:53:04.700782 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.700015 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-kube-api-access-zrtwm" pod "7631819a-de14-4a11-a5a9-9b2e05e7c7a6" (UID: "7631819a-de14-4a11-a5a9-9b2e05e7c7a6"). InnerVolumeSpecName "kube-api-access-zrtwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:53:04.776227 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.776189 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:04.776227 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.776225 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:04.776466 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.776240 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:04.776466 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.776256 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrtwm\" (UniqueName: \"kubernetes.io/projected/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-kube-api-access-zrtwm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:04.776466 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.776269 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:04.776466 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.776280 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:04.776466 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.776291 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:04.776466 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.776303 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:04.776466 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:04.776316 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7631819a-de14-4a11-a5a9-9b2e05e7c7a6-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:05.324743 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:05.324699 4910 generic.go:356] "Generic (PLEG): container finished" podID="7631819a-de14-4a11-a5a9-9b2e05e7c7a6" containerID="f67913e4013c7eead267489dbcc8a1556ee654c25b9ebefa9d6bb63dd6592bd0" exitCode=0 Oct 31 19:53:05.325012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:05.324801 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" event={"ID":"7631819a-de14-4a11-a5a9-9b2e05e7c7a6","Type":"ContainerDied","Data":"f67913e4013c7eead267489dbcc8a1556ee654c25b9ebefa9d6bb63dd6592bd0"} Oct 31 19:53:05.325012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:05.324853 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" event={"ID":"7631819a-de14-4a11-a5a9-9b2e05e7c7a6","Type":"ContainerDied","Data":"87bc256af108967066615400c2a7608bf2c2ab87b62f94da2ae9c5dbbb694c44"} Oct 31 19:53:05.325012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:05.324881 4910 scope.go:122] "RemoveContainer" containerID="f67913e4013c7eead267489dbcc8a1556ee654c25b9ebefa9d6bb63dd6592bd0" Oct 31 19:53:05.325196 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:05.325176 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb" Oct 31 19:53:05.354264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:05.354236 4910 scope.go:122] "RemoveContainer" containerID="de17dab6e975433bab5a8de52bfc85dda1d4fa6131db47c3d80ed5afce1ab072" Oct 31 19:53:05.367527 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:05.367503 4910 scope.go:122] "RemoveContainer" containerID="f67913e4013c7eead267489dbcc8a1556ee654c25b9ebefa9d6bb63dd6592bd0" Oct 31 19:53:05.368283 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:53:05.367960 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f67913e4013c7eead267489dbcc8a1556ee654c25b9ebefa9d6bb63dd6592bd0\": container with ID starting with f67913e4013c7eead267489dbcc8a1556ee654c25b9ebefa9d6bb63dd6592bd0 not found: ID does not exist" containerID="f67913e4013c7eead267489dbcc8a1556ee654c25b9ebefa9d6bb63dd6592bd0" Oct 31 19:53:05.368283 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:05.368003 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67913e4013c7eead267489dbcc8a1556ee654c25b9ebefa9d6bb63dd6592bd0"} err="failed to get container status \"f67913e4013c7eead267489dbcc8a1556ee654c25b9ebefa9d6bb63dd6592bd0\": rpc error: code = NotFound desc = could not find container \"f67913e4013c7eead267489dbcc8a1556ee654c25b9ebefa9d6bb63dd6592bd0\": container with ID starting with f67913e4013c7eead267489dbcc8a1556ee654c25b9ebefa9d6bb63dd6592bd0 not found: ID does not exist" Oct 31 19:53:05.368283 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:05.368048 4910 scope.go:122] "RemoveContainer" containerID="de17dab6e975433bab5a8de52bfc85dda1d4fa6131db47c3d80ed5afce1ab072" Oct 31 19:53:05.368473 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:53:05.368443 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de17dab6e975433bab5a8de52bfc85dda1d4fa6131db47c3d80ed5afce1ab072\": container with ID starting with de17dab6e975433bab5a8de52bfc85dda1d4fa6131db47c3d80ed5afce1ab072 not found: ID does not exist" containerID="de17dab6e975433bab5a8de52bfc85dda1d4fa6131db47c3d80ed5afce1ab072" Oct 31 19:53:05.368516 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:05.368484 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de17dab6e975433bab5a8de52bfc85dda1d4fa6131db47c3d80ed5afce1ab072"} err="failed to get container status \"de17dab6e975433bab5a8de52bfc85dda1d4fa6131db47c3d80ed5afce1ab072\": rpc error: code = NotFound desc = could not find container \"de17dab6e975433bab5a8de52bfc85dda1d4fa6131db47c3d80ed5afce1ab072\": container with ID starting with de17dab6e975433bab5a8de52bfc85dda1d4fa6131db47c3d80ed5afce1ab072 not found: ID does not exist" Oct 31 19:53:05.375955 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:05.374177 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb"] Oct 31 19:53:05.380486 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:05.380283 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-785d26-openshift-default-74465b794c-qg2rb"] Oct 31 19:53:05.803229 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:05.803146 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7631819a-de14-4a11-a5a9-9b2e05e7c7a6" path="/var/lib/kubelet/pods/7631819a-de14-4a11-a5a9-9b2e05e7c7a6/volumes" Oct 31 19:53:12.715929 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:53:12.715871 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:53:19.608015 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:19.607968 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8"] Oct 31 19:53:19.634036 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:19.633987 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d8c398-cqbb8"] Oct 31 19:53:19.796642 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:19.796598 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8814ab6-b8e3-44b3-a145-d16a6b128e49" path="/var/lib/kubelet/pods/f8814ab6-b8e3-44b3-a145-d16a6b128e49/volumes" Oct 31 19:53:26.388878 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:26.388817 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:53:29.427006 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:29.426971 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5"] Oct 31 19:53:29.427753 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:29.427720 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" podUID="27663054-5fbb-4a52-b101-682623a5f518" containerName="istio-proxy" containerID="cri-o://ff6908f3775a8a99746ba2197dd51d1ab4ddac11d84b04ad757ec521ca4a3fde" gracePeriod=30 Oct 31 19:53:34.114333 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:34.114296 4910 patch_prober.go:29] interesting pod/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:53:34.114750 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:34.114353 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" podUID="27663054-5fbb-4a52-b101-682623a5f518" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:53:34.788294 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:34.788207 4910 generic.go:356] "Generic (PLEG): container finished" podID="27663054-5fbb-4a52-b101-682623a5f518" containerID="ff6908f3775a8a99746ba2197dd51d1ab4ddac11d84b04ad757ec521ca4a3fde" exitCode=0 Oct 31 19:53:34.788294 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:34.788280 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" event={"ID":"27663054-5fbb-4a52-b101-682623a5f518","Type":"ContainerDied","Data":"ff6908f3775a8a99746ba2197dd51d1ab4ddac11d84b04ad757ec521ca4a3fde"} Oct 31 19:53:34.788497 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:34.788357 4910 scope.go:122] "RemoveContainer" containerID="cd393767ffcf2b3f30129e3923bb28f97ad899ff30836b6c05c36f1dd0d7f536" Oct 31 19:53:35.229050 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.229016 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:53:35.344379 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.344347 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-workload-socket\") pod \"27663054-5fbb-4a52-b101-682623a5f518\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " Oct 31 19:53:35.344579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.344405 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-workload-certs\") pod \"27663054-5fbb-4a52-b101-682623a5f518\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " Oct 31 19:53:35.344579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.344438 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-credential-socket\") pod \"27663054-5fbb-4a52-b101-682623a5f518\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " Oct 31 19:53:35.344579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.344498 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27663054-5fbb-4a52-b101-682623a5f518-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27663054-5fbb-4a52-b101-682623a5f518-istiod-ca-cert\") pod \"27663054-5fbb-4a52-b101-682623a5f518\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " Oct 31 19:53:35.344579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.344574 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-istio-data\") pod \"27663054-5fbb-4a52-b101-682623a5f518\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " Oct 31 19:53:35.344824 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.344656 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27663054-5fbb-4a52-b101-682623a5f518-kube-api-access-r9kjr\" (UniqueName: \"kubernetes.io/projected/27663054-5fbb-4a52-b101-682623a5f518-kube-api-access-r9kjr\") pod \"27663054-5fbb-4a52-b101-682623a5f518\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " Oct 31 19:53:35.344824 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.344681 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-workload-socket" pod "27663054-5fbb-4a52-b101-682623a5f518" (UID: "27663054-5fbb-4a52-b101-682623a5f518"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:35.344824 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.344708 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27663054-5fbb-4a52-b101-682623a5f518-istio-token\" (UniqueName: \"kubernetes.io/projected/27663054-5fbb-4a52-b101-682623a5f518-istio-token\") pod \"27663054-5fbb-4a52-b101-682623a5f518\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " Oct 31 19:53:35.344824 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.344781 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-workload-certs" pod "27663054-5fbb-4a52-b101-682623a5f518" (UID: "27663054-5fbb-4a52-b101-682623a5f518"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:35.344824 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.344797 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-credential-socket" pod "27663054-5fbb-4a52-b101-682623a5f518" (UID: "27663054-5fbb-4a52-b101-682623a5f518"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:35.345152 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.344824 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-istio-envoy\") pod \"27663054-5fbb-4a52-b101-682623a5f518\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " Oct 31 19:53:35.345152 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.344889 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27663054-5fbb-4a52-b101-682623a5f518-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27663054-5fbb-4a52-b101-682623a5f518-istio-podinfo\") pod \"27663054-5fbb-4a52-b101-682623a5f518\" (UID: \"27663054-5fbb-4a52-b101-682623a5f518\") " Oct 31 19:53:35.345152 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.345043 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27663054-5fbb-4a52-b101-682623a5f518-istiod-ca-cert" pod "27663054-5fbb-4a52-b101-682623a5f518" (UID: "27663054-5fbb-4a52-b101-682623a5f518"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:53:35.348188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.345430 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-istio-data" pod "27663054-5fbb-4a52-b101-682623a5f518" (UID: "27663054-5fbb-4a52-b101-682623a5f518"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:35.348188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.348101 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:35.348188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.348125 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:35.348188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.348139 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:35.348188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.348153 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27663054-5fbb-4a52-b101-682623a5f518-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:35.348188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.348165 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:35.348767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.348743 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27663054-5fbb-4a52-b101-682623a5f518-kube-api-access-r9kjr" pod "27663054-5fbb-4a52-b101-682623a5f518" (UID: "27663054-5fbb-4a52-b101-682623a5f518"). InnerVolumeSpecName "kube-api-access-r9kjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:53:35.348869 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.348780 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27663054-5fbb-4a52-b101-682623a5f518-istio-token" pod "27663054-5fbb-4a52-b101-682623a5f518" (UID: "27663054-5fbb-4a52-b101-682623a5f518"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:53:35.349361 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.349342 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27663054-5fbb-4a52-b101-682623a5f518-istio-podinfo" pod "27663054-5fbb-4a52-b101-682623a5f518" (UID: "27663054-5fbb-4a52-b101-682623a5f518"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:53:35.350116 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.350087 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-istio-envoy" pod "27663054-5fbb-4a52-b101-682623a5f518" (UID: "27663054-5fbb-4a52-b101-682623a5f518"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:53:35.449598 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.449561 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9kjr\" (UniqueName: \"kubernetes.io/projected/27663054-5fbb-4a52-b101-682623a5f518-kube-api-access-r9kjr\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:35.449598 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.449598 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27663054-5fbb-4a52-b101-682623a5f518-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:35.449826 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.449612 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27663054-5fbb-4a52-b101-682623a5f518-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:35.449826 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.449625 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27663054-5fbb-4a52-b101-682623a5f518-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:53:35.797419 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.797388 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" Oct 31 19:53:35.797580 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.797430 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5" event={"ID":"27663054-5fbb-4a52-b101-682623a5f518","Type":"ContainerDied","Data":"885bb4eb6cebd5f642d3dd20cddc9c316deab2c08d56811156aa6f3b05c197a5"} Oct 31 19:53:35.797580 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.797488 4910 scope.go:122] "RemoveContainer" containerID="ff6908f3775a8a99746ba2197dd51d1ab4ddac11d84b04ad757ec521ca4a3fde" Oct 31 19:53:35.820998 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.820968 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5"] Oct 31 19:53:35.827453 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:35.827428 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-jgxn5"] Oct 31 19:53:37.762478 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:37.762437 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27663054-5fbb-4a52-b101-682623a5f518" path="/var/lib/kubelet/pods/27663054-5fbb-4a52-b101-682623a5f518/volumes" Oct 31 19:53:38.548763 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.548722 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h"] Oct 31 19:53:38.549771 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.549744 4910 cpu_manager.go:414] "containerMap: removing container" podUID="7631819a-de14-4a11-a5a9-9b2e05e7c7a6" containerName="istio-proxy" Oct 31 19:53:38.549771 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.549771 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7631819a-de14-4a11-a5a9-9b2e05e7c7a6" containerName="istio-proxy" Oct 31 19:53:38.550070 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.549846 4910 cpu_manager.go:414] "containerMap: removing container" podUID="27663054-5fbb-4a52-b101-682623a5f518" containerName="istio-proxy" Oct 31 19:53:38.550070 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.549854 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27663054-5fbb-4a52-b101-682623a5f518" containerName="istio-proxy" Oct 31 19:53:38.550070 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.549871 4910 cpu_manager.go:414] "containerMap: removing container" podUID="7631819a-de14-4a11-a5a9-9b2e05e7c7a6" containerName="istio-proxy" Oct 31 19:53:38.550070 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.549878 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7631819a-de14-4a11-a5a9-9b2e05e7c7a6" containerName="istio-proxy" Oct 31 19:53:38.550070 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.549932 4910 cpu_manager.go:414] "containerMap: removing container" podUID="27663054-5fbb-4a52-b101-682623a5f518" containerName="istio-proxy" Oct 31 19:53:38.550070 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.549940 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27663054-5fbb-4a52-b101-682623a5f518" containerName="istio-proxy" Oct 31 19:53:38.550070 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.549951 4910 cpu_manager.go:414] "containerMap: removing container" podUID="f8814ab6-b8e3-44b3-a145-d16a6b128e49" containerName="revoke-keys" Oct 31 19:53:38.550070 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.549958 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8814ab6-b8e3-44b3-a145-d16a6b128e49" containerName="revoke-keys" Oct 31 19:53:38.550467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.550134 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="27663054-5fbb-4a52-b101-682623a5f518" containerName="istio-proxy" Oct 31 19:53:38.550467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.550149 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8814ab6-b8e3-44b3-a145-d16a6b128e49" containerName="revoke-keys" Oct 31 19:53:38.550467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.550162 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="7631819a-de14-4a11-a5a9-9b2e05e7c7a6" containerName="istio-proxy" Oct 31 19:53:38.556238 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.556195 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.565859 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.565828 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h"] Oct 31 19:53:38.573221 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.572124 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-a891ce-openshift-default-dockercfg-t8vjd\"" Oct 31 19:53:38.616871 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.614236 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-data\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.616871 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.614329 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvnxv\" (UniqueName: \"kubernetes.io/projected/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-kube-api-access-fvnxv\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.616871 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.614434 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-token\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.616871 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.614545 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-podinfo\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.616871 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.614616 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-envoy\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.616871 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.614681 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-workload-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.616871 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.614747 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-workload-certs\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.616871 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.614798 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-credential-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.616871 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.614854 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istiod-ca-cert\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.716868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.716828 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-workload-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.717078 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.716936 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-workload-certs\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.717078 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.716980 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-credential-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.717078 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.717029 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istiod-ca-cert\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.717272 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.717092 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-data\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.717341 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.717286 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-workload-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.717403 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.717341 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvnxv\" (UniqueName: \"kubernetes.io/projected/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-kube-api-access-fvnxv\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.717403 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.717360 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-workload-certs\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.717403 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.717388 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-credential-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.717515 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.717456 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-data\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.717515 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.717471 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-token\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.717613 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.717592 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-podinfo\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.717696 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.717679 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-envoy\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.717838 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.717817 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istiod-ca-cert\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.720453 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.720422 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-podinfo\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.720654 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.720630 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-envoy\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.731107 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.731078 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-token\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.731438 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.731417 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvnxv\" (UniqueName: \"kubernetes.io/projected/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-kube-api-access-fvnxv\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:38.888042 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:38.888010 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:39.217983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:39.209474 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h"] Oct 31 19:53:39.218352 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:53:39.218324 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89529dc6_ab2a_4b03_a4a9_56f43c12fc31.slice/crio-9577db6fb80ab53931d13b4515dcec571083607e518a9985774492c1fe163ef4 WatchSource:0}: Error finding container 9577db6fb80ab53931d13b4515dcec571083607e518a9985774492c1fe163ef4: Status 404 returned error can't find the container with id 9577db6fb80ab53931d13b4515dcec571083607e518a9985774492c1fe163ef4 Oct 31 19:53:39.221982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:39.221936 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:53:39.222098 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:39.222026 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:53:39.222098 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:39.222067 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:53:39.881229 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:39.881171 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" event={"ID":"89529dc6-ab2a-4b03-a4a9-56f43c12fc31","Type":"ContainerStarted","Data":"36932c944a0bf80b48e898f2ef30d4cbb8efa678ea14ea96a20efe826b3c82a1"} Oct 31 19:53:39.881439 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:39.881242 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" event={"ID":"89529dc6-ab2a-4b03-a4a9-56f43c12fc31","Type":"ContainerStarted","Data":"9577db6fb80ab53931d13b4515dcec571083607e518a9985774492c1fe163ef4"} Oct 31 19:53:39.888254 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:39.888224 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:39.915142 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:39.915084 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" podStartSLOduration=1.915070238 podStartE2EDuration="1.915070238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:53:39.909275087 +0000 UTC m=+2054.272095458" watchObservedRunningTime="2026-10-31 19:53:39.915070238 +0000 UTC m=+2054.277890610" Oct 31 19:53:40.890291 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:40.890260 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 19:53:40.890786 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:40.890309 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" podUID="89529dc6-ab2a-4b03-a4a9-56f43c12fc31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 19:53:41.888928 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:41.888878 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 19:53:41.889107 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:41.888949 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" podUID="89529dc6-ab2a-4b03-a4a9-56f43c12fc31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 19:53:42.717254 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:53:42.717212 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:53:42.888652 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:42.888615 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 19:53:42.888825 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:42.888677 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" podUID="89529dc6-ab2a-4b03-a4a9-56f43c12fc31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 19:53:43.889136 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:43.889101 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 19:53:43.889622 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:43.889162 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" podUID="89529dc6-ab2a-4b03-a4a9-56f43c12fc31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 19:53:44.888658 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:44.888619 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 19:53:44.888841 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:44.888675 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" podUID="89529dc6-ab2a-4b03-a4a9-56f43c12fc31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 19:53:45.889090 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:45.889049 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 19:53:45.889575 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:45.889116 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" podUID="89529dc6-ab2a-4b03-a4a9-56f43c12fc31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 19:53:46.532278 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.532237 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29"] Oct 31 19:53:46.533476 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.533453 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="27663054-5fbb-4a52-b101-682623a5f518" containerName="istio-proxy" Oct 31 19:53:46.533548 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.533488 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="7631819a-de14-4a11-a5a9-9b2e05e7c7a6" containerName="istio-proxy" Oct 31 19:53:46.538142 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.538114 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.551142 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.551112 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-435728-openshift-default-dockercfg-sdb9g\"" Oct 31 19:53:46.568215 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.562198 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29"] Oct 31 19:53:46.620202 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.620164 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-workload-certs\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.620402 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.620238 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-token\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.620402 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.620340 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-podinfo\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.620402 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.620378 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-847qx\" (UniqueName: \"kubernetes.io/projected/f1917335-8922-4b66-a5c4-ec463de8e3ca-kube-api-access-847qx\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.620587 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.620465 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1917335-8922-4b66-a5c4-ec463de8e3ca-istiod-ca-cert\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.620587 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.620534 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-credential-socket\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.620703 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.620652 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-data\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.620703 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.620683 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-envoy\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.620813 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.620715 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-workload-socket\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.733934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.729425 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-workload-certs\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.733934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.729518 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-token\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.733934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.729548 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-podinfo\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.733934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.729571 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-847qx\" (UniqueName: \"kubernetes.io/projected/f1917335-8922-4b66-a5c4-ec463de8e3ca-kube-api-access-847qx\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.733934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.729612 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1917335-8922-4b66-a5c4-ec463de8e3ca-istiod-ca-cert\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.733934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.729716 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-credential-socket\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.733934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.729859 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-data\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.733934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.729907 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-envoy\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.733934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.729977 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-workload-socket\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.733934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.730473 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-workload-socket\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.733934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.730702 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-workload-certs\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.736411 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.736381 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1917335-8922-4b66-a5c4-ec463de8e3ca-istiod-ca-cert\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.737279 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.737255 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-data\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.738176 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.737651 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-credential-socket\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.744885 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.744853 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-envoy\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.765230 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.765182 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-podinfo\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.768418 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.768388 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-token\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.783893 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.783812 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-847qx\" (UniqueName: \"kubernetes.io/projected/f1917335-8922-4b66-a5c4-ec463de8e3ca-kube-api-access-847qx\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.874371 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.874332 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:46.895341 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.895312 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:46.895854 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.895834 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:46.896772 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:46.896750 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 19:53:47.186406 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:53:47.186369 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1917335_8922_4b66_a5c4_ec463de8e3ca.slice/crio-48e74265626f88861c89a9d80a385a9e6333a65317bba47fb3d02d209dbed9d9 WatchSource:0}: Error finding container 48e74265626f88861c89a9d80a385a9e6333a65317bba47fb3d02d209dbed9d9: Status 404 returned error can't find the container with id 48e74265626f88861c89a9d80a385a9e6333a65317bba47fb3d02d209dbed9d9 Oct 31 19:53:47.188793 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:47.188757 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29"] Oct 31 19:53:47.189346 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:47.189282 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:53:47.189452 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:47.189360 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:53:47.189452 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:47.189394 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:53:48.061828 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.061789 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" event={"ID":"f1917335-8922-4b66-a5c4-ec463de8e3ca","Type":"ContainerStarted","Data":"61780a0fbfc07598cfcac6d015ab903cfcf8b87e0cac08f11fae503401e0f683"} Oct 31 19:53:48.062347 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.061858 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" event={"ID":"f1917335-8922-4b66-a5c4-ec463de8e3ca","Type":"ContainerStarted","Data":"48e74265626f88861c89a9d80a385a9e6333a65317bba47fb3d02d209dbed9d9"} Oct 31 19:53:48.091709 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.091653 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" podStartSLOduration=2.091637657 podStartE2EDuration="2.091637657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:53:48.089704614 +0000 UTC m=+2062.452524985" watchObservedRunningTime="2026-10-31 19:53:48.091637657 +0000 UTC m=+2062.454458029" Oct 31 19:53:48.594119 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.593392 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs"] Oct 31 19:53:48.599477 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.599452 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.605369 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.605341 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-e1d060-openshift-default-dockercfg-w5qst\"" Oct 31 19:53:48.610353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.610331 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs"] Oct 31 19:53:48.689848 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.689808 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-data\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.690068 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.689878 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-workload-socket\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.690068 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.689936 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-credential-socket\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.690068 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.690001 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-token\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.690198 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.690071 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-envoy\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.690240 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.690200 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f470386-3ce5-46c1-9c84-68eba9905d6a-istiod-ca-cert\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.690973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.690253 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-podinfo\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.691117 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.691099 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwkr5\" (UniqueName: \"kubernetes.io/projected/4f470386-3ce5-46c1-9c84-68eba9905d6a-kube-api-access-zwkr5\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.691169 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.691144 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-workload-certs\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.793105 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.793062 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-data\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.793267 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.793129 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-workload-socket\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.793267 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.793165 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-credential-socket\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.793267 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.793229 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-token\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.793413 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.793266 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-envoy\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.793413 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.793355 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f470386-3ce5-46c1-9c84-68eba9905d6a-istiod-ca-cert\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.793486 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.793428 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-podinfo\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.793580 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.793565 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwkr5\" (UniqueName: \"kubernetes.io/projected/4f470386-3ce5-46c1-9c84-68eba9905d6a-kube-api-access-zwkr5\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.793619 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.793605 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-workload-certs\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.793664 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.793608 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-workload-socket\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.794407 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.794379 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f470386-3ce5-46c1-9c84-68eba9905d6a-istiod-ca-cert\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.794536 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.794437 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-workload-certs\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.794596 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.794556 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-data\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.794692 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.794673 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-credential-socket\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.796232 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.796205 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-envoy\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.796810 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.796790 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-podinfo\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.810659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.810628 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwkr5\" (UniqueName: \"kubernetes.io/projected/4f470386-3ce5-46c1-9c84-68eba9905d6a-kube-api-access-zwkr5\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.810973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.810948 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-token\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:48.874944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.874822 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:48.876955 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.876901 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 31 19:53:48.877058 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.876980 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" podUID="f1917335-8922-4b66-a5c4-ec463de8e3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 31 19:53:48.952883 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:48.952837 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:49.272387 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:53:49.272343 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f470386_3ce5_46c1_9c84_68eba9905d6a.slice/crio-8c2fa99f2a54aaca7e26b4d94e7c737600303f8354fecc87c8348f52d7eb2dfb WatchSource:0}: Error finding container 8c2fa99f2a54aaca7e26b4d94e7c737600303f8354fecc87c8348f52d7eb2dfb: Status 404 returned error can't find the container with id 8c2fa99f2a54aaca7e26b4d94e7c737600303f8354fecc87c8348f52d7eb2dfb Oct 31 19:53:49.276209 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:49.276175 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs"] Oct 31 19:53:49.276443 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:49.276402 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:53:49.276546 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:49.276490 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:53:49.276546 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:49.276522 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:53:49.874780 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:49.874732 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 31 19:53:49.875028 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:49.874815 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" podUID="f1917335-8922-4b66-a5c4-ec463de8e3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 31 19:53:50.089354 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:50.089309 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" event={"ID":"4f470386-3ce5-46c1-9c84-68eba9905d6a","Type":"ContainerStarted","Data":"8be4abb8922c9e19636bcb8b07e453982265b9bf3a32629f4469babecb96ee5c"} Oct 31 19:53:50.089527 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:50.089384 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" event={"ID":"4f470386-3ce5-46c1-9c84-68eba9905d6a","Type":"ContainerStarted","Data":"8c2fa99f2a54aaca7e26b4d94e7c737600303f8354fecc87c8348f52d7eb2dfb"} Oct 31 19:53:50.117026 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:50.116952 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" podStartSLOduration=2.116935244 podStartE2EDuration="2.116935244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:53:50.11498748 +0000 UTC m=+2064.477807853" watchObservedRunningTime="2026-10-31 19:53:50.116935244 +0000 UTC m=+2064.479755617" Oct 31 19:53:50.875421 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:50.875386 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 31 19:53:50.876068 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:50.876027 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" podUID="f1917335-8922-4b66-a5c4-ec463de8e3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 31 19:53:50.955628 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:50.955592 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:50.961046 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:50.961011 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 31 19:53:50.961339 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:50.961310 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" podUID="4f470386-3ce5-46c1-9c84-68eba9905d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 31 19:53:50.991057 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:50.991019 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8"] Oct 31 19:53:50.996876 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:50.996851 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.005336 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.005256 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-c55cf1-openshift-default-dockercfg-jg5ls\"" Oct 31 19:53:51.011512 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.011370 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8"] Oct 31 19:53:51.096018 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.090825 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-workload-socket\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.096018 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.091655 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-workload-certs\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.096018 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.091739 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-credential-socket\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.096018 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.091790 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-envoy\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.096018 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.091823 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-data\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.194628 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.194435 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-token\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.194628 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.194496 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-data\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.194628 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.194611 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-workload-socket\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.194923 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.194879 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-workload-certs\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.194986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.194939 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istiod-ca-cert\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.195059 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.194993 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5fq4\" (UniqueName: \"kubernetes.io/projected/15e605d1-e3b9-4c0b-9b03-689c480dd89c-kube-api-access-l5fq4\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.195651 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.195429 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-data\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.195778 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.195758 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-workload-certs\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.195851 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.195823 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-workload-socket\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.195953 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.195884 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-credential-socket\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.196197 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.196162 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-credential-socket\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.196378 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.196360 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-podinfo\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.196652 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.196633 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-envoy\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.202467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.201597 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-envoy\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.354934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.350610 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-token\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.354934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.351082 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istiod-ca-cert\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.354934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.351148 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5fq4\" (UniqueName: \"kubernetes.io/projected/15e605d1-e3b9-4c0b-9b03-689c480dd89c-kube-api-access-l5fq4\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.354934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.351216 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-podinfo\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.362938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.359480 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istiod-ca-cert\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.429935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.429776 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5fq4\" (UniqueName: \"kubernetes.io/projected/15e605d1-e3b9-4c0b-9b03-689c480dd89c-kube-api-access-l5fq4\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.430324 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.430286 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-token\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.430997 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.430971 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-podinfo\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.622945 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.622217 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:51.892980 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.874840 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 31 19:53:51.892980 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.874936 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" podUID="f1917335-8922-4b66-a5c4-ec463de8e3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 31 19:53:51.988954 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.984587 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 31 19:53:51.988954 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:51.984664 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" podUID="4f470386-3ce5-46c1-9c84-68eba9905d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 31 19:53:52.206227 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:52.206192 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8"] Oct 31 19:53:52.210660 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:52.210625 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:53:52.210768 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:52.210705 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:53:52.210768 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:52.210753 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:53:52.882937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:52.882482 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 31 19:53:52.882937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:52.882543 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" podUID="f1917335-8922-4b66-a5c4-ec463de8e3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 31 19:53:52.960938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:52.960077 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 31 19:53:52.960938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:52.960153 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" podUID="4f470386-3ce5-46c1-9c84-68eba9905d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 31 19:53:53.228991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:53.228342 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" event={"ID":"15e605d1-e3b9-4c0b-9b03-689c480dd89c","Type":"ContainerStarted","Data":"ad8bda9ae5e5870b65111ae1b0a4b5d912f159710737aa492fe40a8bb81bbb57"} Oct 31 19:53:53.228991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:53.228416 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" event={"ID":"15e605d1-e3b9-4c0b-9b03-689c480dd89c","Type":"ContainerStarted","Data":"19e03dca9ecd33ed64066e697c92732065b9e1d727d55074cc081560271d4a2a"} Oct 31 19:53:53.268937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:53.268837 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" podStartSLOduration=3.268818002 podStartE2EDuration="3.268818002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:53:53.262385687 +0000 UTC m=+2067.625206059" watchObservedRunningTime="2026-10-31 19:53:53.268818002 +0000 UTC m=+2067.631638375" Oct 31 19:53:53.636935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:53.636640 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:53.640933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:53.638531 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 31 19:53:53.640933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:53.638591 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" podUID="15e605d1-e3b9-4c0b-9b03-689c480dd89c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 31 19:53:53.887001 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:53.886074 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 31 19:53:53.887001 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:53.886158 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" podUID="f1917335-8922-4b66-a5c4-ec463de8e3ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 31 19:53:53.973695 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:53.973634 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 31 19:53:53.974388 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:53.974350 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" podUID="4f470386-3ce5-46c1-9c84-68eba9905d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 31 19:53:54.645936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:54.645337 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 31 19:53:54.645936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:54.645399 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" podUID="15e605d1-e3b9-4c0b-9b03-689c480dd89c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 31 19:53:54.889238 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:54.889204 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:54.890245 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:54.890218 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:54.892473 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:54.892440 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 19:53:54.956098 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:54.956001 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 31 19:53:54.956368 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:54.956332 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" podUID="4f470386-3ce5-46c1-9c84-68eba9905d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 31 19:53:55.632337 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:55.632297 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 31 19:53:55.632927 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:55.632880 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" podUID="15e605d1-e3b9-4c0b-9b03-689c480dd89c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 31 19:53:55.955436 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:55.955336 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body= Oct 31 19:53:55.956098 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:55.956061 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" podUID="4f470386-3ce5-46c1-9c84-68eba9905d6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" Oct 31 19:53:56.633335 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:56.633297 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 31 19:53:56.633992 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:56.633954 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" podUID="15e605d1-e3b9-4c0b-9b03-689c480dd89c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 31 19:53:56.959332 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:56.959251 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:56.959803 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:56.959779 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:56.960815 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:56.960792 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:53:57.634471 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:57.634017 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 31 19:53:57.634471 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:57.634083 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" podUID="15e605d1-e3b9-4c0b-9b03-689c480dd89c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 31 19:53:58.635373 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:58.635333 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 31 19:53:58.636074 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:58.636024 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" podUID="15e605d1-e3b9-4c0b-9b03-689c480dd89c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 31 19:53:59.634154 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:59.634118 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:59.634656 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:59.634635 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:53:59.635509 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:53:59.635486 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 19:54:12.717396 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:54:12.717355 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:54:26.403541 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:26.403511 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 19:54:26.407365 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:26.407338 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 19:54:30.461132 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:30.461083 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:54:30.461572 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:30.461179 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" podUID="64efa0c3-abc5-4253-8f01-cb9954e33dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:54:33.842270 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:33.842236 4910 generic.go:356] "Generic (PLEG): container finished" podID="64efa0c3-abc5-4253-8f01-cb9954e33dc6" containerID="f916b44d93c7fa872d1ae3cf94864bdcb7690d0977fabdc11b5e37bad0588312" exitCode=0 Oct 31 19:54:33.842766 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:33.842307 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" event={"ID":"64efa0c3-abc5-4253-8f01-cb9954e33dc6","Type":"ContainerDied","Data":"f916b44d93c7fa872d1ae3cf94864bdcb7690d0977fabdc11b5e37bad0588312"} Oct 31 19:54:33.842832 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:33.842776 4910 scope.go:122] "RemoveContainer" containerID="f916b44d93c7fa872d1ae3cf94864bdcb7690d0977fabdc11b5e37bad0588312" Oct 31 19:54:33.844478 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:33.844439 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:54:33.844615 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:33.844534 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:54:33.844615 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:33.844588 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:54:34.857350 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:34.857316 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" event={"ID":"64efa0c3-abc5-4253-8f01-cb9954e33dc6","Type":"ContainerStarted","Data":"e722a46bfbd56c97023369be32ecbd1a3382095ec2e5c1d9c950aff77459dc7d"} Oct 31 19:54:35.196803 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:35.196714 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:54:35.196999 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:35.196979 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 31 19:54:35.197065 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:35.197021 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" podUID="64efa0c3-abc5-4253-8f01-cb9954e33dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 31 19:54:36.197290 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:36.197211 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 31 19:54:36.197290 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:36.197277 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" podUID="64efa0c3-abc5-4253-8f01-cb9954e33dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 31 19:54:37.197395 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:37.197356 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 31 19:54:37.197803 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:37.197424 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" podUID="64efa0c3-abc5-4253-8f01-cb9954e33dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 31 19:54:38.199866 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:38.197394 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 31 19:54:38.199866 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:38.197460 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" podUID="64efa0c3-abc5-4253-8f01-cb9954e33dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 31 19:54:39.197744 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:39.197706 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 31 19:54:39.197965 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:39.197764 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" podUID="64efa0c3-abc5-4253-8f01-cb9954e33dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 31 19:54:40.197663 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:40.197622 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" start-of-body= Oct 31 19:54:40.198184 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:40.197682 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" podUID="64efa0c3-abc5-4253-8f01-cb9954e33dc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.214:15021/healthz/ready\": dial tcp 10.132.0.214:15021: connect: connection refused" Oct 31 19:54:41.205093 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:41.205061 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:54:41.205587 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:41.205572 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:54:41.206415 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:54:41.206393 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 19:54:42.715713 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:54:42.715670 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:55:10.473548 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:10.473422 4910 generic.go:356] "Generic (PLEG): container finished" podID="949b61a5-f07e-47c1-b360-a3bfde012f42" containerID="77cc12100e7999d67458362233f0ea318c168ef7b636d85a2cdef5eafe159689" exitCode=0 Oct 31 19:55:10.473548 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:10.473498 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" event={"ID":"949b61a5-f07e-47c1-b360-a3bfde012f42","Type":"ContainerDied","Data":"77cc12100e7999d67458362233f0ea318c168ef7b636d85a2cdef5eafe159689"} Oct 31 19:55:10.474064 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:10.474035 4910 scope.go:122] "RemoveContainer" containerID="77cc12100e7999d67458362233f0ea318c168ef7b636d85a2cdef5eafe159689" Oct 31 19:55:10.476206 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:10.476169 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:55:10.476345 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:10.476274 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:55:10.476345 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:10.476322 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:55:11.560007 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:11.559831 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" event={"ID":"949b61a5-f07e-47c1-b360-a3bfde012f42","Type":"ContainerStarted","Data":"771ee87af4ec83f3266431bf0447e56a1d0d18820ff12724e8d48d0c100dabac"} Oct 31 19:55:12.550043 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:12.550006 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:55:12.550457 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:12.550424 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 31 19:55:12.550607 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:12.550499 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" podUID="949b61a5-f07e-47c1-b360-a3bfde012f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 31 19:55:12.716537 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:55:12.716491 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:55:13.550004 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:13.547839 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 31 19:55:13.550004 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:13.547899 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" podUID="949b61a5-f07e-47c1-b360-a3bfde012f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 31 19:55:14.548494 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:14.548455 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 31 19:55:14.549931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:14.549873 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" podUID="949b61a5-f07e-47c1-b360-a3bfde012f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 31 19:55:15.548329 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:15.548287 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 31 19:55:15.548654 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:15.548622 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" podUID="949b61a5-f07e-47c1-b360-a3bfde012f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 31 19:55:16.549406 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:16.549358 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 31 19:55:16.549995 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:16.549438 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" podUID="949b61a5-f07e-47c1-b360-a3bfde012f42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 31 19:55:17.552822 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:17.552787 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:55:17.553369 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:17.553352 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:55:17.554242 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:17.554217 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 19:55:24.044294 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:24.044257 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cefa93db-2v5jk"] Oct 31 19:55:24.053617 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:24.053588 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cefa93db-2v5jk"] Oct 31 19:55:25.252140 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:25.252095 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bcc1e0f-sl8bc"] Oct 31 19:55:25.257286 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:25.257265 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc1e0f-sl8bc" Oct 31 19:55:25.263161 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:25.263106 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:55:25.263339 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:25.263180 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:55:25.264808 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:25.264787 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bcc1e0f-sl8bc"] Oct 31 19:55:25.310664 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:25.310617 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121d4267-2283-43ab-8181-dd231d3350bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bcc1e0f-sl8bc\" (UID: \"121d4267-2283-43ab-8181-dd231d3350bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc1e0f-sl8bc" Oct 31 19:55:25.310854 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:25.310789 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121d4267-2283-43ab-8181-dd231d3350bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bcc1e0f-sl8bc\" (UID: \"121d4267-2283-43ab-8181-dd231d3350bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc1e0f-sl8bc" Oct 31 19:55:25.310964 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:25.310897 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dm7v\" (UniqueName: \"kubernetes.io/projected/121d4267-2283-43ab-8181-dd231d3350bb-kube-api-access-7dm7v\") pod \"maas-api-revoke-sub-5bcc1e0f-sl8bc\" (UID: \"121d4267-2283-43ab-8181-dd231d3350bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc1e0f-sl8bc" Oct 31 19:55:25.424934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:25.413384 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121d4267-2283-43ab-8181-dd231d3350bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bcc1e0f-sl8bc\" (UID: \"121d4267-2283-43ab-8181-dd231d3350bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc1e0f-sl8bc" Oct 31 19:55:25.443823 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:25.443785 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121d4267-2283-43ab-8181-dd231d3350bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bcc1e0f-sl8bc\" (UID: \"121d4267-2283-43ab-8181-dd231d3350bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc1e0f-sl8bc" Oct 31 19:55:25.444311 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:25.444282 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dm7v\" (UniqueName: \"kubernetes.io/projected/121d4267-2283-43ab-8181-dd231d3350bb-kube-api-access-7dm7v\") pod \"maas-api-revoke-sub-5bcc1e0f-sl8bc\" (UID: \"121d4267-2283-43ab-8181-dd231d3350bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc1e0f-sl8bc" Oct 31 19:55:25.444528 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:25.444504 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121d4267-2283-43ab-8181-dd231d3350bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bcc1e0f-sl8bc\" (UID: \"121d4267-2283-43ab-8181-dd231d3350bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc1e0f-sl8bc" Oct 31 19:55:25.444613 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:25.435580 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121d4267-2283-43ab-8181-dd231d3350bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bcc1e0f-sl8bc\" (UID: \"121d4267-2283-43ab-8181-dd231d3350bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc1e0f-sl8bc" Oct 31 19:55:25.459188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:25.459140 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dm7v\" (UniqueName: \"kubernetes.io/projected/121d4267-2283-43ab-8181-dd231d3350bb-kube-api-access-7dm7v\") pod \"maas-api-revoke-sub-5bcc1e0f-sl8bc\" (UID: \"121d4267-2283-43ab-8181-dd231d3350bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc1e0f-sl8bc" Oct 31 19:55:25.583355 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:25.583314 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc1e0f-sl8bc" Oct 31 19:55:25.758387 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:25.758351 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1363646-d9cf-4748-a70d-76daf00aba42" path="/var/lib/kubelet/pods/c1363646-d9cf-4748-a70d-76daf00aba42/volumes" Oct 31 19:55:25.873398 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:55:25.873359 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod121d4267_2283_43ab_8181_dd231d3350bb.slice/crio-3cd91b4764bef53f0c7617f52835a42d126cb262ce37dac85747aa5aacc3dcc9 WatchSource:0}: Error finding container 3cd91b4764bef53f0c7617f52835a42d126cb262ce37dac85747aa5aacc3dcc9: Status 404 returned error can't find the container with id 3cd91b4764bef53f0c7617f52835a42d126cb262ce37dac85747aa5aacc3dcc9 Oct 31 19:55:25.876802 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:25.876769 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bcc1e0f-sl8bc"] Oct 31 19:55:26.729450 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:26.729406 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc1e0f-sl8bc" event={"ID":"121d4267-2283-43ab-8181-dd231d3350bb","Type":"ContainerStarted","Data":"7b4454198ba5114c1cfffcf978f97b69540e07ca2a01681c3d17ab904d9d8051"} Oct 31 19:55:26.729860 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:26.729479 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc1e0f-sl8bc" event={"ID":"121d4267-2283-43ab-8181-dd231d3350bb","Type":"ContainerStarted","Data":"3cd91b4764bef53f0c7617f52835a42d126cb262ce37dac85747aa5aacc3dcc9"} Oct 31 19:55:26.751731 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:26.751676 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc1e0f-sl8bc" podStartSLOduration=1.751661573 podStartE2EDuration="1.751661573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:55:26.749665263 +0000 UTC m=+2161.112485651" watchObservedRunningTime="2026-10-31 19:55:26.751661573 +0000 UTC m=+2161.114481944" Oct 31 19:55:27.741338 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:27.741297 4910 generic.go:356] "Generic (PLEG): container finished" podID="121d4267-2283-43ab-8181-dd231d3350bb" containerID="7b4454198ba5114c1cfffcf978f97b69540e07ca2a01681c3d17ab904d9d8051" exitCode=0 Oct 31 19:55:27.741731 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:27.741341 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc1e0f-sl8bc" event={"ID":"121d4267-2283-43ab-8181-dd231d3350bb","Type":"ContainerDied","Data":"7b4454198ba5114c1cfffcf978f97b69540e07ca2a01681c3d17ab904d9d8051"} Oct 31 19:55:28.928295 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:28.928260 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc1e0f-sl8bc" Oct 31 19:55:29.034181 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:29.034100 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/121d4267-2283-43ab-8181-dd231d3350bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121d4267-2283-43ab-8181-dd231d3350bb-maas-api-xks-ca\") pod \"121d4267-2283-43ab-8181-dd231d3350bb\" (UID: \"121d4267-2283-43ab-8181-dd231d3350bb\") " Oct 31 19:55:29.034181 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:29.034170 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/121d4267-2283-43ab-8181-dd231d3350bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121d4267-2283-43ab-8181-dd231d3350bb-maas-api-openshift-ca\") pod \"121d4267-2283-43ab-8181-dd231d3350bb\" (UID: \"121d4267-2283-43ab-8181-dd231d3350bb\") " Oct 31 19:55:29.034400 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:29.034302 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/121d4267-2283-43ab-8181-dd231d3350bb-kube-api-access-7dm7v\" (UniqueName: \"kubernetes.io/projected/121d4267-2283-43ab-8181-dd231d3350bb-kube-api-access-7dm7v\") pod \"121d4267-2283-43ab-8181-dd231d3350bb\" (UID: \"121d4267-2283-43ab-8181-dd231d3350bb\") " Oct 31 19:55:29.034579 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:29.034551 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/121d4267-2283-43ab-8181-dd231d3350bb-maas-api-openshift-ca" pod "121d4267-2283-43ab-8181-dd231d3350bb" (UID: "121d4267-2283-43ab-8181-dd231d3350bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:55:29.035019 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:29.034997 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121d4267-2283-43ab-8181-dd231d3350bb-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:55:29.036442 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:29.036415 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/121d4267-2283-43ab-8181-dd231d3350bb-maas-api-xks-ca" pod "121d4267-2283-43ab-8181-dd231d3350bb" (UID: "121d4267-2283-43ab-8181-dd231d3350bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:55:29.036549 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:29.036456 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/121d4267-2283-43ab-8181-dd231d3350bb-kube-api-access-7dm7v" pod "121d4267-2283-43ab-8181-dd231d3350bb" (UID: "121d4267-2283-43ab-8181-dd231d3350bb"). InnerVolumeSpecName "kube-api-access-7dm7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:55:29.136747 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:29.136717 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dm7v\" (UniqueName: \"kubernetes.io/projected/121d4267-2283-43ab-8181-dd231d3350bb-kube-api-access-7dm7v\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:55:29.136747 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:29.136746 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121d4267-2283-43ab-8181-dd231d3350bb-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:55:29.762519 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:29.762486 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc1e0f-sl8bc" Oct 31 19:55:29.762519 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:29.762502 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bcc1e0f-sl8bc" event={"ID":"121d4267-2283-43ab-8181-dd231d3350bb","Type":"ContainerDied","Data":"3cd91b4764bef53f0c7617f52835a42d126cb262ce37dac85747aa5aacc3dcc9"} Oct 31 19:55:29.762776 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:29.762573 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cd91b4764bef53f0c7617f52835a42d126cb262ce37dac85747aa5aacc3dcc9" Oct 31 19:55:32.780860 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:32.780826 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z"] Oct 31 19:55:32.781359 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:32.781077 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" podUID="7031809a-4dff-4444-bd5c-521501265431" containerName="discovery" containerID="cri-o://c59dd42a41180198b2e8d88811ce2a90bc69574b8562de2df31275bd96dafd31" gracePeriod=30 Oct 31 19:55:33.112966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.112939 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:55:33.208383 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.208344 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7031809a-4dff-4444-bd5c-521501265431-istio-token\" (UniqueName: \"kubernetes.io/projected/7031809a-4dff-4444-bd5c-521501265431-istio-token\") pod \"7031809a-4dff-4444-bd5c-521501265431\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " Oct 31 19:55:33.208593 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.208408 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7031809a-4dff-4444-bd5c-521501265431-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7031809a-4dff-4444-bd5c-521501265431-local-certs\") pod \"7031809a-4dff-4444-bd5c-521501265431\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " Oct 31 19:55:33.208593 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.208505 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7031809a-4dff-4444-bd5c-521501265431-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7031809a-4dff-4444-bd5c-521501265431-istio-kubeconfig\") pod \"7031809a-4dff-4444-bd5c-521501265431\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " Oct 31 19:55:33.208593 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.208587 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7031809a-4dff-4444-bd5c-521501265431-kube-api-access-tctf9\" (UniqueName: \"kubernetes.io/projected/7031809a-4dff-4444-bd5c-521501265431-kube-api-access-tctf9\") pod \"7031809a-4dff-4444-bd5c-521501265431\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " Oct 31 19:55:33.208775 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.208662 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7031809a-4dff-4444-bd5c-521501265431-cacerts\" (UniqueName: \"kubernetes.io/secret/7031809a-4dff-4444-bd5c-521501265431-cacerts\") pod \"7031809a-4dff-4444-bd5c-521501265431\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " Oct 31 19:55:33.208836 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.208821 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7031809a-4dff-4444-bd5c-521501265431-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7031809a-4dff-4444-bd5c-521501265431-istio-csr-ca-configmap\") pod \"7031809a-4dff-4444-bd5c-521501265431\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " Oct 31 19:55:33.208897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.208865 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7031809a-4dff-4444-bd5c-521501265431-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7031809a-4dff-4444-bd5c-521501265431-istio-csr-dns-cert\") pod \"7031809a-4dff-4444-bd5c-521501265431\" (UID: \"7031809a-4dff-4444-bd5c-521501265431\") " Oct 31 19:55:33.210353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.210307 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7031809a-4dff-4444-bd5c-521501265431-istio-csr-ca-configmap" pod "7031809a-4dff-4444-bd5c-521501265431" (UID: "7031809a-4dff-4444-bd5c-521501265431"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:55:33.211957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.211308 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7031809a-4dff-4444-bd5c-521501265431-istio-token" pod "7031809a-4dff-4444-bd5c-521501265431" (UID: "7031809a-4dff-4444-bd5c-521501265431"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:55:33.212143 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.212115 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7031809a-4dff-4444-bd5c-521501265431-cacerts" pod "7031809a-4dff-4444-bd5c-521501265431" (UID: "7031809a-4dff-4444-bd5c-521501265431"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:55:33.214274 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.214237 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7031809a-4dff-4444-bd5c-521501265431-local-certs" pod "7031809a-4dff-4444-bd5c-521501265431" (UID: "7031809a-4dff-4444-bd5c-521501265431"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:55:33.214418 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.214393 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7031809a-4dff-4444-bd5c-521501265431-istio-kubeconfig" pod "7031809a-4dff-4444-bd5c-521501265431" (UID: "7031809a-4dff-4444-bd5c-521501265431"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:55:33.214643 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.214619 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7031809a-4dff-4444-bd5c-521501265431-istio-csr-dns-cert" pod "7031809a-4dff-4444-bd5c-521501265431" (UID: "7031809a-4dff-4444-bd5c-521501265431"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:55:33.214718 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.214658 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7031809a-4dff-4444-bd5c-521501265431-kube-api-access-tctf9" pod "7031809a-4dff-4444-bd5c-521501265431" (UID: "7031809a-4dff-4444-bd5c-521501265431"). InnerVolumeSpecName "kube-api-access-tctf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:55:33.311562 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.311525 4910 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7031809a-4dff-4444-bd5c-521501265431-istio-csr-dns-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:55:33.311562 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.311558 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7031809a-4dff-4444-bd5c-521501265431-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:55:33.311562 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.311568 4910 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7031809a-4dff-4444-bd5c-521501265431-local-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:55:33.311829 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.311577 4910 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7031809a-4dff-4444-bd5c-521501265431-istio-kubeconfig\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:55:33.311829 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.311585 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tctf9\" (UniqueName: \"kubernetes.io/projected/7031809a-4dff-4444-bd5c-521501265431-kube-api-access-tctf9\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:55:33.311829 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.311595 4910 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7031809a-4dff-4444-bd5c-521501265431-cacerts\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:55:33.311829 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.311603 4910 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7031809a-4dff-4444-bd5c-521501265431-istio-csr-ca-configmap\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:55:33.801629 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.801591 4910 generic.go:356] "Generic (PLEG): container finished" podID="7031809a-4dff-4444-bd5c-521501265431" containerID="c59dd42a41180198b2e8d88811ce2a90bc69574b8562de2df31275bd96dafd31" exitCode=0 Oct 31 19:55:33.802137 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.801651 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" event={"ID":"7031809a-4dff-4444-bd5c-521501265431","Type":"ContainerDied","Data":"c59dd42a41180198b2e8d88811ce2a90bc69574b8562de2df31275bd96dafd31"} Oct 31 19:55:33.802137 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.801675 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" Oct 31 19:55:33.802137 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.801692 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z" event={"ID":"7031809a-4dff-4444-bd5c-521501265431","Type":"ContainerDied","Data":"00755ba6e121432771c8a1c63719f0f44f4100df72eb1e483cf795a8c76b448b"} Oct 31 19:55:33.802137 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.801711 4910 scope.go:122] "RemoveContainer" containerID="c59dd42a41180198b2e8d88811ce2a90bc69574b8562de2df31275bd96dafd31" Oct 31 19:55:33.814196 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.814169 4910 scope.go:122] "RemoveContainer" containerID="c59dd42a41180198b2e8d88811ce2a90bc69574b8562de2df31275bd96dafd31" Oct 31 19:55:33.814491 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:55:33.814470 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c59dd42a41180198b2e8d88811ce2a90bc69574b8562de2df31275bd96dafd31\": container with ID starting with c59dd42a41180198b2e8d88811ce2a90bc69574b8562de2df31275bd96dafd31 not found: ID does not exist" containerID="c59dd42a41180198b2e8d88811ce2a90bc69574b8562de2df31275bd96dafd31" Oct 31 19:55:33.814582 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.814500 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c59dd42a41180198b2e8d88811ce2a90bc69574b8562de2df31275bd96dafd31"} err="failed to get container status \"c59dd42a41180198b2e8d88811ce2a90bc69574b8562de2df31275bd96dafd31\": rpc error: code = NotFound desc = could not find container \"c59dd42a41180198b2e8d88811ce2a90bc69574b8562de2df31275bd96dafd31\": container with ID starting with c59dd42a41180198b2e8d88811ce2a90bc69574b8562de2df31275bd96dafd31 not found: ID does not exist" Oct 31 19:55:33.835864 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.835830 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z"] Oct 31 19:55:33.840646 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:33.840617 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l4p8z"] Oct 31 19:55:34.119704 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.119660 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8"] Oct 31 19:55:34.120677 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.120651 4910 cpu_manager.go:414] "containerMap: removing container" podUID="7031809a-4dff-4444-bd5c-521501265431" containerName="discovery" Oct 31 19:55:34.120677 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.120677 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7031809a-4dff-4444-bd5c-521501265431" containerName="discovery" Oct 31 19:55:34.120889 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.120695 4910 cpu_manager.go:414] "containerMap: removing container" podUID="121d4267-2283-43ab-8181-dd231d3350bb" containerName="revoke-keys" Oct 31 19:55:34.120889 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.120703 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="121d4267-2283-43ab-8181-dd231d3350bb" containerName="revoke-keys" Oct 31 19:55:34.121060 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.120959 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="7031809a-4dff-4444-bd5c-521501265431" containerName="discovery" Oct 31 19:55:34.121060 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.120978 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="121d4267-2283-43ab-8181-dd231d3350bb" containerName="revoke-keys" Oct 31 19:55:34.126715 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.126688 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.157395 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.157360 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8"] Oct 31 19:55:34.229696 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.229652 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89wnr\" (UniqueName: \"kubernetes.io/projected/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-kube-api-access-89wnr\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.229696 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.229716 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.230002 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.229927 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.230085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.230061 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.230185 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.230168 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.230233 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.230206 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.230372 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.230354 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.230450 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.230422 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.230569 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.230473 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.332294 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.332259 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.332506 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.332320 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.332506 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.332371 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89wnr\" (UniqueName: \"kubernetes.io/projected/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-kube-api-access-89wnr\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.332633 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.332566 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.332722 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.332703 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.332813 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.332739 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.332813 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.332780 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.332966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.332823 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.332966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.332867 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.332966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.332903 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.332966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.332957 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.333187 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.333069 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.333420 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.333399 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.333553 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.333532 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.335278 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.335246 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.335513 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.335494 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.345268 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.345220 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89wnr\" (UniqueName: \"kubernetes.io/projected/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-kube-api-access-89wnr\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.346957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.346931 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.443682 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.443589 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:34.813693 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.813660 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8"] Oct 31 19:55:34.814149 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:55:34.813893 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb49720_62f4_4d08_bad3_51b33ab7fa7f.slice/crio-67ad32e32456fd9ef242e9e8bb3f4df3ec07f9dcf12db40bc21afa7b883442b4 WatchSource:0}: Error finding container 67ad32e32456fd9ef242e9e8bb3f4df3ec07f9dcf12db40bc21afa7b883442b4: Status 404 returned error can't find the container with id 67ad32e32456fd9ef242e9e8bb3f4df3ec07f9dcf12db40bc21afa7b883442b4 Oct 31 19:55:34.817321 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.817287 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:55:34.817411 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.817357 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:55:34.817411 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:34.817387 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:55:35.761128 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:35.761089 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7031809a-4dff-4444-bd5c-521501265431" path="/var/lib/kubelet/pods/7031809a-4dff-4444-bd5c-521501265431/volumes" Oct 31 19:55:35.852935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:35.849167 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" event={"ID":"5eb49720-62f4-4d08-bad3-51b33ab7fa7f","Type":"ContainerStarted","Data":"2b67d9321bb77d833c5c4707e6a48cbe598eb73c445fa093c94d40d99f044b41"} Oct 31 19:55:35.852935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:35.849251 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" event={"ID":"5eb49720-62f4-4d08-bad3-51b33ab7fa7f","Type":"ContainerStarted","Data":"67ad32e32456fd9ef242e9e8bb3f4df3ec07f9dcf12db40bc21afa7b883442b4"} Oct 31 19:55:35.880012 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:35.879956 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" podStartSLOduration=1.8799409950000001 podStartE2EDuration="1.879940995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:55:35.876627632 +0000 UTC m=+2170.239448004" watchObservedRunningTime="2026-10-31 19:55:35.879940995 +0000 UTC m=+2170.242761437" Oct 31 19:55:36.448937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:36.448295 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:36.454933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:36.454032 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 19:55:36.454933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:36.454104 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" podUID="5eb49720-62f4-4d08-bad3-51b33ab7fa7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 19:55:36.702330 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:36.702257 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e63e490b-p2ljk"] Oct 31 19:55:36.707083 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:36.707059 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e490b-p2ljk" Oct 31 19:55:36.718270 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:36.718235 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:55:36.742632 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:36.742593 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:55:36.815002 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:36.814967 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e63e490b-p2ljk"] Oct 31 19:55:36.856937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:36.824307 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/087fb9af-3c00-4e5c-bdba-9f485e3be7ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e63e490b-p2ljk\" (UID: \"087fb9af-3c00-4e5c-bdba-9f485e3be7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e490b-p2ljk" Oct 31 19:55:36.865061 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:36.865020 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/087fb9af-3c00-4e5c-bdba-9f485e3be7ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e63e490b-p2ljk\" (UID: \"087fb9af-3c00-4e5c-bdba-9f485e3be7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e490b-p2ljk" Oct 31 19:55:36.865408 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:36.865389 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d47n\" (UniqueName: \"kubernetes.io/projected/087fb9af-3c00-4e5c-bdba-9f485e3be7ec-kube-api-access-6d47n\") pod \"maas-api-revoke-sub-e63e490b-p2ljk\" (UID: \"087fb9af-3c00-4e5c-bdba-9f485e3be7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e490b-p2ljk" Oct 31 19:55:36.969716 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:36.967978 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/087fb9af-3c00-4e5c-bdba-9f485e3be7ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e63e490b-p2ljk\" (UID: \"087fb9af-3c00-4e5c-bdba-9f485e3be7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e490b-p2ljk" Oct 31 19:55:36.969716 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:36.968176 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/087fb9af-3c00-4e5c-bdba-9f485e3be7ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e63e490b-p2ljk\" (UID: \"087fb9af-3c00-4e5c-bdba-9f485e3be7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e490b-p2ljk" Oct 31 19:55:36.969716 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:36.968298 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d47n\" (UniqueName: \"kubernetes.io/projected/087fb9af-3c00-4e5c-bdba-9f485e3be7ec-kube-api-access-6d47n\") pod \"maas-api-revoke-sub-e63e490b-p2ljk\" (UID: \"087fb9af-3c00-4e5c-bdba-9f485e3be7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e490b-p2ljk" Oct 31 19:55:36.969716 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:36.969336 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/087fb9af-3c00-4e5c-bdba-9f485e3be7ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e63e490b-p2ljk\" (UID: \"087fb9af-3c00-4e5c-bdba-9f485e3be7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e490b-p2ljk" Oct 31 19:55:36.972595 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:36.972571 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/087fb9af-3c00-4e5c-bdba-9f485e3be7ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e63e490b-p2ljk\" (UID: \"087fb9af-3c00-4e5c-bdba-9f485e3be7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e490b-p2ljk" Oct 31 19:55:36.982238 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:36.982219 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d47n\" (UniqueName: \"kubernetes.io/projected/087fb9af-3c00-4e5c-bdba-9f485e3be7ec-kube-api-access-6d47n\") pod \"maas-api-revoke-sub-e63e490b-p2ljk\" (UID: \"087fb9af-3c00-4e5c-bdba-9f485e3be7ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e490b-p2ljk" Oct 31 19:55:37.116809 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:37.116776 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e490b-p2ljk" Oct 31 19:55:37.459440 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:37.459042 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e63e490b-p2ljk"] Oct 31 19:55:37.460750 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:37.460616 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 19:55:37.461076 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:37.460704 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" podUID="5eb49720-62f4-4d08-bad3-51b33ab7fa7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 19:55:37.899727 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:37.899683 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e490b-p2ljk" event={"ID":"087fb9af-3c00-4e5c-bdba-9f485e3be7ec","Type":"ContainerStarted","Data":"23c4aa1d67a9a5b84f8469d9f62fc543336be1daf1f43089310d9551766450e5"} Oct 31 19:55:38.447353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:38.447316 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 19:55:38.447664 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:38.447635 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" podUID="5eb49720-62f4-4d08-bad3-51b33ab7fa7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 19:55:38.911595 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:38.911556 4910 generic.go:356] "Generic (PLEG): container finished" podID="087fb9af-3c00-4e5c-bdba-9f485e3be7ec" containerID="3ebe502147ee80ed5b815939868fa8a7c0e358de4d30da7be1ad2806eeba84b2" exitCode=0 Oct 31 19:55:38.912085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:38.911615 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e490b-p2ljk" event={"ID":"087fb9af-3c00-4e5c-bdba-9f485e3be7ec","Type":"ContainerDied","Data":"3ebe502147ee80ed5b815939868fa8a7c0e358de4d30da7be1ad2806eeba84b2"} Oct 31 19:55:39.449346 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:39.449304 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 19:55:39.449562 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:39.449367 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" podUID="5eb49720-62f4-4d08-bad3-51b33ab7fa7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 19:55:40.207137 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:40.207068 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e490b-p2ljk" Oct 31 19:55:40.261962 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:40.261893 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/087fb9af-3c00-4e5c-bdba-9f485e3be7ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/087fb9af-3c00-4e5c-bdba-9f485e3be7ec-maas-api-openshift-ca\") pod \"087fb9af-3c00-4e5c-bdba-9f485e3be7ec\" (UID: \"087fb9af-3c00-4e5c-bdba-9f485e3be7ec\") " Oct 31 19:55:40.262159 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:40.262010 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/087fb9af-3c00-4e5c-bdba-9f485e3be7ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/087fb9af-3c00-4e5c-bdba-9f485e3be7ec-maas-api-xks-ca\") pod \"087fb9af-3c00-4e5c-bdba-9f485e3be7ec\" (UID: \"087fb9af-3c00-4e5c-bdba-9f485e3be7ec\") " Oct 31 19:55:40.262159 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:40.262040 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/087fb9af-3c00-4e5c-bdba-9f485e3be7ec-kube-api-access-6d47n\" (UniqueName: \"kubernetes.io/projected/087fb9af-3c00-4e5c-bdba-9f485e3be7ec-kube-api-access-6d47n\") pod \"087fb9af-3c00-4e5c-bdba-9f485e3be7ec\" (UID: \"087fb9af-3c00-4e5c-bdba-9f485e3be7ec\") " Oct 31 19:55:40.263468 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:40.263437 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/087fb9af-3c00-4e5c-bdba-9f485e3be7ec-maas-api-openshift-ca" pod "087fb9af-3c00-4e5c-bdba-9f485e3be7ec" (UID: "087fb9af-3c00-4e5c-bdba-9f485e3be7ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:55:40.270514 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:40.270481 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/087fb9af-3c00-4e5c-bdba-9f485e3be7ec-kube-api-access-6d47n" pod "087fb9af-3c00-4e5c-bdba-9f485e3be7ec" (UID: "087fb9af-3c00-4e5c-bdba-9f485e3be7ec"). InnerVolumeSpecName "kube-api-access-6d47n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:55:40.272472 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:40.272444 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/087fb9af-3c00-4e5c-bdba-9f485e3be7ec-maas-api-xks-ca" pod "087fb9af-3c00-4e5c-bdba-9f485e3be7ec" (UID: "087fb9af-3c00-4e5c-bdba-9f485e3be7ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:55:40.366632 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:40.365220 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/087fb9af-3c00-4e5c-bdba-9f485e3be7ec-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:55:40.366632 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:40.365252 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/087fb9af-3c00-4e5c-bdba-9f485e3be7ec-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:55:40.366632 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:40.365264 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d47n\" (UniqueName: \"kubernetes.io/projected/087fb9af-3c00-4e5c-bdba-9f485e3be7ec-kube-api-access-6d47n\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:55:40.448940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:40.447319 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 19:55:40.448940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:40.447383 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" podUID="5eb49720-62f4-4d08-bad3-51b33ab7fa7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 19:55:40.974085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:40.974039 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e490b-p2ljk" event={"ID":"087fb9af-3c00-4e5c-bdba-9f485e3be7ec","Type":"ContainerDied","Data":"23c4aa1d67a9a5b84f8469d9f62fc543336be1daf1f43089310d9551766450e5"} Oct 31 19:55:40.974768 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:40.974743 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23c4aa1d67a9a5b84f8469d9f62fc543336be1daf1f43089310d9551766450e5" Oct 31 19:55:40.974943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:40.974241 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e63e490b-p2ljk" Oct 31 19:55:41.444922 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:41.444870 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body= Oct 31 19:55:41.445452 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:41.444961 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" podUID="5eb49720-62f4-4d08-bad3-51b33ab7fa7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" Oct 31 19:55:42.449158 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:42.449127 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:42.449646 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:42.449536 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:42.450442 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:42.450407 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 19:55:42.745829 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:55:42.743103 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:55:43.031068 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:43.030981 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea7fff1d-8kmst"] Oct 31 19:55:43.035950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:43.035890 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea7fff1d-8kmst"] Oct 31 19:55:43.760831 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:43.760793 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d44421ef-3970-47ca-8407-f2e75c7b9372" path="/var/lib/kubelet/pods/d44421ef-3970-47ca-8407-f2e75c7b9372/volumes" Oct 31 19:55:48.775938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:48.775492 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm"] Oct 31 19:55:48.785947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:48.776533 4910 cpu_manager.go:414] "containerMap: removing container" podUID="087fb9af-3c00-4e5c-bdba-9f485e3be7ec" containerName="revoke-keys" Oct 31 19:55:48.785947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:48.776558 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="087fb9af-3c00-4e5c-bdba-9f485e3be7ec" containerName="revoke-keys" Oct 31 19:55:48.785947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:48.776755 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="087fb9af-3c00-4e5c-bdba-9f485e3be7ec" containerName="revoke-keys" Oct 31 19:55:48.799934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:48.787570 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm" Oct 31 19:55:48.810968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:48.804925 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:55:48.855197 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:48.846631 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:55:48.868337 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:48.868198 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm"] Oct 31 19:55:48.873138 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:48.873081 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b302d7-624e-4af7-81ca-8383abf99498-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm\" (UID: \"61b302d7-624e-4af7-81ca-8383abf99498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm" Oct 31 19:55:48.873462 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:48.873428 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx9tj\" (UniqueName: \"kubernetes.io/projected/61b302d7-624e-4af7-81ca-8383abf99498-kube-api-access-cx9tj\") pod \"maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm\" (UID: \"61b302d7-624e-4af7-81ca-8383abf99498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm" Oct 31 19:55:48.873771 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:48.873752 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b302d7-624e-4af7-81ca-8383abf99498-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm\" (UID: \"61b302d7-624e-4af7-81ca-8383abf99498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm" Oct 31 19:55:48.975923 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:48.975884 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b302d7-624e-4af7-81ca-8383abf99498-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm\" (UID: \"61b302d7-624e-4af7-81ca-8383abf99498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm" Oct 31 19:55:48.976120 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:48.976058 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b302d7-624e-4af7-81ca-8383abf99498-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm\" (UID: \"61b302d7-624e-4af7-81ca-8383abf99498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm" Oct 31 19:55:48.976179 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:48.976124 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx9tj\" (UniqueName: \"kubernetes.io/projected/61b302d7-624e-4af7-81ca-8383abf99498-kube-api-access-cx9tj\") pod \"maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm\" (UID: \"61b302d7-624e-4af7-81ca-8383abf99498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm" Oct 31 19:55:48.976724 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:48.976697 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b302d7-624e-4af7-81ca-8383abf99498-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm\" (UID: \"61b302d7-624e-4af7-81ca-8383abf99498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm" Oct 31 19:55:48.978480 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:48.978460 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b302d7-624e-4af7-81ca-8383abf99498-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm\" (UID: \"61b302d7-624e-4af7-81ca-8383abf99498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm" Oct 31 19:55:48.990835 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:48.990807 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx9tj\" (UniqueName: \"kubernetes.io/projected/61b302d7-624e-4af7-81ca-8383abf99498-kube-api-access-cx9tj\") pod \"maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm\" (UID: \"61b302d7-624e-4af7-81ca-8383abf99498\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm" Oct 31 19:55:49.146855 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.146818 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm"] Oct 31 19:55:49.153154 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.152596 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.177393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.177358 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm"] Oct 31 19:55:49.178527 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.178323 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm" Oct 31 19:55:49.281641 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.281239 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.281641 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.281334 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.281641 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.281373 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.281641 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.281404 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.281641 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.281463 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.281641 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.281521 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.281641 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.281579 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.282181 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.281678 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.282181 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.281851 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8h26\" (UniqueName: \"kubernetes.io/projected/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-kube-api-access-j8h26\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.384138 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.384099 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.384335 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.384213 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.384335 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.384259 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.384335 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.384306 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.384507 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.384388 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.384507 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.384469 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.384631 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.384560 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.384631 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.384612 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.384747 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.384667 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8h26\" (UniqueName: \"kubernetes.io/projected/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-kube-api-access-j8h26\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.384807 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.384771 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.384899 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.384877 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.385199 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.385174 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.385246 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.385198 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.385426 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.385410 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.386651 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.386628 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.387146 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.387125 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.406951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.406858 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.409852 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.409825 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8h26\" (UniqueName: \"kubernetes.io/projected/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-kube-api-access-j8h26\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.487177 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:55:49.487147 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b302d7_624e_4af7_81ca_8383abf99498.slice/crio-0e64d3fc94a5594291b3d150925902b6636cd6a2d997587e70f5a5944d72a0d8 WatchSource:0}: Error finding container 0e64d3fc94a5594291b3d150925902b6636cd6a2d997587e70f5a5944d72a0d8: Status 404 returned error can't find the container with id 0e64d3fc94a5594291b3d150925902b6636cd6a2d997587e70f5a5944d72a0d8 Oct 31 19:55:49.488680 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.488654 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm"] Oct 31 19:55:49.490695 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.490674 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:49.802375 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:55:49.802339 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf41d2753_d5fe_4a08_a4e4_180c1c3c5972.slice/crio-e3f9fd8374eb4ae15c72202497b4961e85996e67cd72238980bfea39637386c4 WatchSource:0}: Error finding container e3f9fd8374eb4ae15c72202497b4961e85996e67cd72238980bfea39637386c4: Status 404 returned error can't find the container with id e3f9fd8374eb4ae15c72202497b4961e85996e67cd72238980bfea39637386c4 Oct 31 19:55:49.804463 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.804436 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm"] Oct 31 19:55:49.805471 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.805432 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:55:49.805572 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.805518 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:55:49.805572 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:49.805566 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:55:50.081804 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:50.081754 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" event={"ID":"f41d2753-d5fe-4a08-a4e4-180c1c3c5972","Type":"ContainerStarted","Data":"34872d8e140d477d4e1acaff2892d5365efd05027a5e8755d308114fc0503dae"} Oct 31 19:55:50.082025 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:50.081823 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" event={"ID":"f41d2753-d5fe-4a08-a4e4-180c1c3c5972","Type":"ContainerStarted","Data":"e3f9fd8374eb4ae15c72202497b4961e85996e67cd72238980bfea39637386c4"} Oct 31 19:55:50.083768 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:50.083742 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm" event={"ID":"61b302d7-624e-4af7-81ca-8383abf99498","Type":"ContainerStarted","Data":"44d2f91e56c72b22210a545fa6abe640c7440d95c70b521b23b6a00ea845f6d9"} Oct 31 19:55:50.083927 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:50.083783 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm" event={"ID":"61b302d7-624e-4af7-81ca-8383abf99498","Type":"ContainerStarted","Data":"0e64d3fc94a5594291b3d150925902b6636cd6a2d997587e70f5a5944d72a0d8"} Oct 31 19:55:50.106944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:50.106860 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" podStartSLOduration=1.106841873 podStartE2EDuration="1.106841873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:55:50.105527189 +0000 UTC m=+2184.468347560" watchObservedRunningTime="2026-10-31 19:55:50.106841873 +0000 UTC m=+2184.469662247" Oct 31 19:55:50.129474 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:50.129412 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm" podStartSLOduration=2.129395291 podStartE2EDuration="2.129395291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:55:50.124375545 +0000 UTC m=+2184.487195914" watchObservedRunningTime="2026-10-31 19:55:50.129395291 +0000 UTC m=+2184.492215665" Oct 31 19:55:50.491615 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:50.491536 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:50.493533 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:50.493508 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 19:55:50.493658 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:50.493555 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" podUID="f41d2753-d5fe-4a08-a4e4-180c1c3c5972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 19:55:51.093927 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:51.093880 4910 generic.go:356] "Generic (PLEG): container finished" podID="61b302d7-624e-4af7-81ca-8383abf99498" containerID="44d2f91e56c72b22210a545fa6abe640c7440d95c70b521b23b6a00ea845f6d9" exitCode=0 Oct 31 19:55:51.094313 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:51.093943 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm" event={"ID":"61b302d7-624e-4af7-81ca-8383abf99498","Type":"ContainerDied","Data":"44d2f91e56c72b22210a545fa6abe640c7440d95c70b521b23b6a00ea845f6d9"} Oct 31 19:55:51.491786 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:51.491702 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 19:55:51.491786 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:51.491767 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" podUID="f41d2753-d5fe-4a08-a4e4-180c1c3c5972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 19:55:52.374269 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:52.374243 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm" Oct 31 19:55:52.491394 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:52.491362 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 19:55:52.491570 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:52.491411 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" podUID="f41d2753-d5fe-4a08-a4e4-180c1c3c5972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 19:55:52.493891 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:52.493868 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61b302d7-624e-4af7-81ca-8383abf99498-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b302d7-624e-4af7-81ca-8383abf99498-maas-api-xks-ca\") pod \"61b302d7-624e-4af7-81ca-8383abf99498\" (UID: \"61b302d7-624e-4af7-81ca-8383abf99498\") " Oct 31 19:55:52.494022 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:52.493924 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b302d7-624e-4af7-81ca-8383abf99498-kube-api-access-cx9tj\" (UniqueName: \"kubernetes.io/projected/61b302d7-624e-4af7-81ca-8383abf99498-kube-api-access-cx9tj\") pod \"61b302d7-624e-4af7-81ca-8383abf99498\" (UID: \"61b302d7-624e-4af7-81ca-8383abf99498\") " Oct 31 19:55:52.494247 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:52.494219 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b302d7-624e-4af7-81ca-8383abf99498-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b302d7-624e-4af7-81ca-8383abf99498-maas-api-openshift-ca\") pod \"61b302d7-624e-4af7-81ca-8383abf99498\" (UID: \"61b302d7-624e-4af7-81ca-8383abf99498\") " Oct 31 19:55:52.494610 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:52.494583 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b302d7-624e-4af7-81ca-8383abf99498-maas-api-openshift-ca" pod "61b302d7-624e-4af7-81ca-8383abf99498" (UID: "61b302d7-624e-4af7-81ca-8383abf99498"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:55:52.494952 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:52.494905 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b302d7-624e-4af7-81ca-8383abf99498-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:55:52.496353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:52.496324 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61b302d7-624e-4af7-81ca-8383abf99498-maas-api-xks-ca" pod "61b302d7-624e-4af7-81ca-8383abf99498" (UID: "61b302d7-624e-4af7-81ca-8383abf99498"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:55:52.496452 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:52.496383 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b302d7-624e-4af7-81ca-8383abf99498-kube-api-access-cx9tj" pod "61b302d7-624e-4af7-81ca-8383abf99498" (UID: "61b302d7-624e-4af7-81ca-8383abf99498"). InnerVolumeSpecName "kube-api-access-cx9tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:55:52.596801 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:52.596768 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b302d7-624e-4af7-81ca-8383abf99498-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:55:52.596801 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:52.596801 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx9tj\" (UniqueName: \"kubernetes.io/projected/61b302d7-624e-4af7-81ca-8383abf99498-kube-api-access-cx9tj\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:55:53.112025 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:53.111962 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm" event={"ID":"61b302d7-624e-4af7-81ca-8383abf99498","Type":"ContainerDied","Data":"0e64d3fc94a5594291b3d150925902b6636cd6a2d997587e70f5a5944d72a0d8"} Oct 31 19:55:53.112208 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:53.112041 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e64d3fc94a5594291b3d150925902b6636cd6a2d997587e70f5a5944d72a0d8" Oct 31 19:55:53.112208 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:53.112044 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm" Oct 31 19:55:53.491187 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:53.491100 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 19:55:53.491187 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:53.491154 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" podUID="f41d2753-d5fe-4a08-a4e4-180c1c3c5972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 19:55:54.491927 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:54.491887 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 19:55:54.492421 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:54.491960 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" podUID="f41d2753-d5fe-4a08-a4e4-180c1c3c5972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 19:55:55.054768 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:55.054732 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b85fa0a2-68tnp"] Oct 31 19:55:55.059955 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:55.059902 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b85fa0a2-68tnp"] Oct 31 19:55:55.491353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:55.491312 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 19:55:55.491566 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:55.491388 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" podUID="f41d2753-d5fe-4a08-a4e4-180c1c3c5972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 19:55:55.824084 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:55.824044 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ca271f-5d14-4807-928d-a4c5c23debc4" path="/var/lib/kubelet/pods/76ca271f-5d14-4807-928d-a4c5c23debc4/volumes" Oct 31 19:55:56.496304 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:56.496265 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 19:55:56.496826 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:56.496794 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" podUID="f41d2753-d5fe-4a08-a4e4-180c1c3c5972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 19:55:57.314938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:57.314332 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:55:57.315462 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:57.315352 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:55:57.494890 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:57.494853 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 19:55:57.495230 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:57.495150 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" podUID="f41d2753-d5fe-4a08-a4e4-180c1c3c5972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 19:55:58.496575 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:58.496537 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:58.497078 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:58.497016 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:55:58.497852 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:55:58.497827 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 19:56:04.214994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.214961 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr"] Oct 31 19:56:04.216585 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.216557 4910 cpu_manager.go:414] "containerMap: removing container" podUID="61b302d7-624e-4af7-81ca-8383abf99498" containerName="revoke-keys" Oct 31 19:56:04.216774 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.216758 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b302d7-624e-4af7-81ca-8383abf99498" containerName="revoke-keys" Oct 31 19:56:04.217277 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.217253 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b302d7-624e-4af7-81ca-8383abf99498" containerName="revoke-keys" Oct 31 19:56:04.223513 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.223490 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.248634 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.248607 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr"] Oct 31 19:56:04.375530 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.375489 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.375750 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.375538 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.375750 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.375630 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb11f75-3d05-4e9c-8db7-caa224621cad-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.375750 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.375707 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.375946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.375819 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk2v4\" (UniqueName: \"kubernetes.io/projected/ceb11f75-3d05-4e9c-8db7-caa224621cad-kube-api-access-xk2v4\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.375946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.375865 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.376042 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.375967 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.376081 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.376048 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.376175 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.376160 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.478056 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.477967 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.478224 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.478070 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.478224 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.478102 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.478224 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.478135 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb11f75-3d05-4e9c-8db7-caa224621cad-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.478224 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.478170 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.478443 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.478281 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk2v4\" (UniqueName: \"kubernetes.io/projected/ceb11f75-3d05-4e9c-8db7-caa224621cad-kube-api-access-xk2v4\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.478443 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.478326 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.478443 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.478378 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.478634 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.478474 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.478634 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.478617 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.478875 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.478833 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.478988 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.478876 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.478988 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.478974 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.479079 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.479051 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb11f75-3d05-4e9c-8db7-caa224621cad-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.480537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.480513 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.480950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.480903 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.491682 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.491655 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk2v4\" (UniqueName: \"kubernetes.io/projected/ceb11f75-3d05-4e9c-8db7-caa224621cad-kube-api-access-xk2v4\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.492357 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.492339 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.555425 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.555389 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:04.855022 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.854988 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr"] Oct 31 19:56:04.855703 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:56:04.855671 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb11f75_3d05_4e9c_8db7_caa224621cad.slice/crio-021584c5023e4f58c7d21eba4fe24ae32d43359747184e9e1484be3ea735481d WatchSource:0}: Error finding container 021584c5023e4f58c7d21eba4fe24ae32d43359747184e9e1484be3ea735481d: Status 404 returned error can't find the container with id 021584c5023e4f58c7d21eba4fe24ae32d43359747184e9e1484be3ea735481d Oct 31 19:56:04.859351 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.859293 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:56:04.859485 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.859382 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:56:04.859485 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:04.859429 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:56:05.315123 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:05.315086 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" event={"ID":"ceb11f75-3d05-4e9c-8db7-caa224621cad","Type":"ContainerStarted","Data":"fedbdb204387b3305e30411da5ecfa09c5282cfb45d84d787e225acd28eac397"} Oct 31 19:56:05.315639 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:05.315152 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" event={"ID":"ceb11f75-3d05-4e9c-8db7-caa224621cad","Type":"ContainerStarted","Data":"021584c5023e4f58c7d21eba4fe24ae32d43359747184e9e1484be3ea735481d"} Oct 31 19:56:05.555746 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:05.555709 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:06.558389 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:06.558306 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 31 19:56:06.558389 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:06.558365 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" podUID="ceb11f75-3d05-4e9c-8db7-caa224621cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 31 19:56:07.556917 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:07.556590 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 31 19:56:07.556917 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:07.556644 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" podUID="ceb11f75-3d05-4e9c-8db7-caa224621cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 31 19:56:08.568987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:08.558305 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 31 19:56:08.568987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:08.558372 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" podUID="ceb11f75-3d05-4e9c-8db7-caa224621cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 31 19:56:09.556235 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:09.556188 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 31 19:56:09.556457 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:09.556261 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" podUID="ceb11f75-3d05-4e9c-8db7-caa224621cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 31 19:56:10.556210 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:10.556174 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body= Oct 31 19:56:10.556619 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:10.556242 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" podUID="ceb11f75-3d05-4e9c-8db7-caa224621cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" Oct 31 19:56:11.642223 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:11.642191 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:11.643201 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:11.643170 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:11.644974 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:11.644949 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 19:56:11.673776 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:11.673729 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" podStartSLOduration=7.673715868 podStartE2EDuration="7.673715868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:56:05.349227268 +0000 UTC m=+2199.712047679" watchObservedRunningTime="2026-10-31 19:56:11.673715868 +0000 UTC m=+2206.036536239" Oct 31 19:56:12.794612 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:56:12.794576 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:56:14.129171 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:14.129103 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm"] Oct 31 19:56:14.132077 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:14.132024 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-754e0a-76wzm"] Oct 31 19:56:15.756878 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:15.756841 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61b302d7-624e-4af7-81ca-8383abf99498" path="/var/lib/kubelet/pods/61b302d7-624e-4af7-81ca-8383abf99498/volumes" Oct 31 19:56:23.907380 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:23.907334 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs"] Oct 31 19:56:23.913462 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:23.913430 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:23.940616 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:23.939340 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs"] Oct 31 19:56:23.969264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:23.968233 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-workload-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:23.969264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:23.968419 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctvzx\" (UniqueName: \"kubernetes.io/projected/ce05955e-1925-4469-978f-29f94fc3d6f0-kube-api-access-ctvzx\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:23.969264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:23.968507 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-data\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:23.969264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:23.968538 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-podinfo\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:23.969264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:23.968584 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-token\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:23.969264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:23.968602 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-workload-certs\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:23.969264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:23.968638 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-envoy\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:23.969264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:23.968656 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce05955e-1925-4469-978f-29f94fc3d6f0-istiod-ca-cert\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:23.969264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:23.968719 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-credential-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:24.070431 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.070399 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-workload-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:24.070600 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.070546 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctvzx\" (UniqueName: \"kubernetes.io/projected/ce05955e-1925-4469-978f-29f94fc3d6f0-kube-api-access-ctvzx\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:24.070673 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.070655 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-data\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:24.070715 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.070700 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-podinfo\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:24.070784 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.070761 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-token\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:24.070838 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.070795 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-workload-certs\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:24.070897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.070857 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-workload-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:24.071028 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.070965 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-envoy\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:24.071028 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.071005 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce05955e-1925-4469-978f-29f94fc3d6f0-istiod-ca-cert\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:24.071155 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.071062 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-data\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:24.071155 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.071094 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-credential-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:24.071270 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.071157 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-workload-certs\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:24.071571 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.071544 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-credential-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:24.071789 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.071769 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce05955e-1925-4469-978f-29f94fc3d6f0-istiod-ca-cert\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:24.073222 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.073202 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-envoy\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:24.073536 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.073513 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-podinfo\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:24.084401 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.084353 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctvzx\" (UniqueName: \"kubernetes.io/projected/ce05955e-1925-4469-978f-29f94fc3d6f0-kube-api-access-ctvzx\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:24.084922 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.084892 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-token\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:24.197681 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.196386 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf"] Oct 31 19:56:24.197681 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.196763 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" podUID="1f13c4cd-264c-44fc-814b-9dee39280ca1" containerName="istio-proxy" containerID="cri-o://8d85361589025d9bbf9c8670a78a7d8311566a9db402db386e1e07344ef11211" gracePeriod=30 Oct 31 19:56:24.220046 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.220009 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx"] Oct 31 19:56:24.220326 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.220285 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" podUID="ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08" containerName="istio-proxy" containerID="cri-o://2fbb5c4e654e2ab3c176bc4217fb13c5367740a612ab358e8658572099e62958" gracePeriod=30 Oct 31 19:56:24.225289 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.225259 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk"] Oct 31 19:56:24.225506 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.225480 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" podUID="c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0" containerName="istio-proxy" containerID="cri-o://aba5df7861a9f680491ef00e55f4b40eade180552490970cc19cea0a2cb3733b" gracePeriod=30 Oct 31 19:56:24.244214 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.244182 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:24.853367 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.853331 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm"] Oct 31 19:56:24.858474 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.858449 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm" Oct 31 19:56:24.864310 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.864286 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:56:24.864446 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.864411 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:56:24.870333 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.870308 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm"] Oct 31 19:56:24.880959 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:56:24.880885 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce05955e_1925_4469_978f_29f94fc3d6f0.slice/crio-d4e813422d5d9f357077f03afb88227da08ccdfe08ed1c5913619a2b551c215b WatchSource:0}: Error finding container d4e813422d5d9f357077f03afb88227da08ccdfe08ed1c5913619a2b551c215b: Status 404 returned error can't find the container with id d4e813422d5d9f357077f03afb88227da08ccdfe08ed1c5913619a2b551c215b Oct 31 19:56:24.883922 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.883880 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:56:24.884037 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.884003 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:56:24.884098 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.884036 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:56:24.901099 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.901077 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs"] Oct 31 19:56:24.998251 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.998210 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279b2c3c-1804-4129-9ce6-d8bcfd847096-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm\" (UID: \"279b2c3c-1804-4129-9ce6-d8bcfd847096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm" Oct 31 19:56:24.998676 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.998377 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxdsx\" (UniqueName: \"kubernetes.io/projected/279b2c3c-1804-4129-9ce6-d8bcfd847096-kube-api-access-wxdsx\") pod \"maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm\" (UID: \"279b2c3c-1804-4129-9ce6-d8bcfd847096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm" Oct 31 19:56:24.998676 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:24.998407 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279b2c3c-1804-4129-9ce6-d8bcfd847096-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm\" (UID: \"279b2c3c-1804-4129-9ce6-d8bcfd847096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm" Oct 31 19:56:25.100935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:25.100810 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279b2c3c-1804-4129-9ce6-d8bcfd847096-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm\" (UID: \"279b2c3c-1804-4129-9ce6-d8bcfd847096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm" Oct 31 19:56:25.101142 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:25.100963 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxdsx\" (UniqueName: \"kubernetes.io/projected/279b2c3c-1804-4129-9ce6-d8bcfd847096-kube-api-access-wxdsx\") pod \"maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm\" (UID: \"279b2c3c-1804-4129-9ce6-d8bcfd847096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm" Oct 31 19:56:25.101142 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:25.100999 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279b2c3c-1804-4129-9ce6-d8bcfd847096-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm\" (UID: \"279b2c3c-1804-4129-9ce6-d8bcfd847096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm" Oct 31 19:56:25.101976 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:25.101785 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279b2c3c-1804-4129-9ce6-d8bcfd847096-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm\" (UID: \"279b2c3c-1804-4129-9ce6-d8bcfd847096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm" Oct 31 19:56:25.103964 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:25.103441 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279b2c3c-1804-4129-9ce6-d8bcfd847096-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm\" (UID: \"279b2c3c-1804-4129-9ce6-d8bcfd847096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm" Oct 31 19:56:25.114494 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:25.114454 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxdsx\" (UniqueName: \"kubernetes.io/projected/279b2c3c-1804-4129-9ce6-d8bcfd847096-kube-api-access-wxdsx\") pod \"maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm\" (UID: \"279b2c3c-1804-4129-9ce6-d8bcfd847096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm" Oct 31 19:56:25.182375 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:25.182278 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm" Oct 31 19:56:25.482356 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:56:25.482320 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod279b2c3c_1804_4129_9ce6_d8bcfd847096.slice/crio-b94a6b8f91508b9cde3986e1d31d8ec94c0e680315ade70fed270a7d16700336 WatchSource:0}: Error finding container b94a6b8f91508b9cde3986e1d31d8ec94c0e680315ade70fed270a7d16700336: Status 404 returned error can't find the container with id b94a6b8f91508b9cde3986e1d31d8ec94c0e680315ade70fed270a7d16700336 Oct 31 19:56:25.482689 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:25.482670 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm"] Oct 31 19:56:25.663439 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:25.663394 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" event={"ID":"ce05955e-1925-4469-978f-29f94fc3d6f0","Type":"ContainerStarted","Data":"5a1bb571db833edf9c804ba2527b8fb1e84361d13c52cdd186ee06d716312cb7"} Oct 31 19:56:25.663613 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:25.663466 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" event={"ID":"ce05955e-1925-4469-978f-29f94fc3d6f0","Type":"ContainerStarted","Data":"d4e813422d5d9f357077f03afb88227da08ccdfe08ed1c5913619a2b551c215b"} Oct 31 19:56:25.664451 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:25.664427 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm" event={"ID":"279b2c3c-1804-4129-9ce6-d8bcfd847096","Type":"ContainerStarted","Data":"b94a6b8f91508b9cde3986e1d31d8ec94c0e680315ade70fed270a7d16700336"} Oct 31 19:56:25.692779 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:25.692725 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" podStartSLOduration=2.692711991 podStartE2EDuration="2.692711991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:56:25.688948765 +0000 UTC m=+2220.051769139" watchObservedRunningTime="2026-10-31 19:56:25.692711991 +0000 UTC m=+2220.055532363" Oct 31 19:56:26.249445 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:26.245405 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:26.256543 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:26.253435 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 31 19:56:26.256543 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:26.253502 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" podUID="ce05955e-1925-4469-978f-29f94fc3d6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 31 19:56:26.675978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:26.675905 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm" event={"ID":"279b2c3c-1804-4129-9ce6-d8bcfd847096","Type":"ContainerStarted","Data":"802f0797e77955c3ecab5b994f28424afa35f2dffc5025b674bb77a1bc733a31"} Oct 31 19:56:26.700576 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:26.700528 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm" podStartSLOduration=2.700514337 podStartE2EDuration="2.700514337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:56:26.695152399 +0000 UTC m=+2221.057972773" watchObservedRunningTime="2026-10-31 19:56:26.700514337 +0000 UTC m=+2221.063334741" Oct 31 19:56:27.254939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:27.247351 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 31 19:56:27.254939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:27.247423 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" podUID="ce05955e-1925-4469-978f-29f94fc3d6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 31 19:56:27.688171 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:27.688133 4910 generic.go:356] "Generic (PLEG): container finished" podID="279b2c3c-1804-4129-9ce6-d8bcfd847096" containerID="802f0797e77955c3ecab5b994f28424afa35f2dffc5025b674bb77a1bc733a31" exitCode=0 Oct 31 19:56:27.688366 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:27.688209 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm" event={"ID":"279b2c3c-1804-4129-9ce6-d8bcfd847096","Type":"ContainerDied","Data":"802f0797e77955c3ecab5b994f28424afa35f2dffc5025b674bb77a1bc733a31"} Oct 31 19:56:28.247936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:28.247066 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 31 19:56:28.247936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:28.247131 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" podUID="ce05955e-1925-4469-978f-29f94fc3d6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 31 19:56:28.946936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:28.946170 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm" Oct 31 19:56:29.086790 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.086746 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/279b2c3c-1804-4129-9ce6-d8bcfd847096-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279b2c3c-1804-4129-9ce6-d8bcfd847096-maas-api-xks-ca\") pod \"279b2c3c-1804-4129-9ce6-d8bcfd847096\" (UID: \"279b2c3c-1804-4129-9ce6-d8bcfd847096\") " Oct 31 19:56:29.087210 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.087180 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/279b2c3c-1804-4129-9ce6-d8bcfd847096-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279b2c3c-1804-4129-9ce6-d8bcfd847096-maas-api-openshift-ca\") pod \"279b2c3c-1804-4129-9ce6-d8bcfd847096\" (UID: \"279b2c3c-1804-4129-9ce6-d8bcfd847096\") " Oct 31 19:56:29.087442 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.087420 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/279b2c3c-1804-4129-9ce6-d8bcfd847096-kube-api-access-wxdsx\" (UniqueName: \"kubernetes.io/projected/279b2c3c-1804-4129-9ce6-d8bcfd847096-kube-api-access-wxdsx\") pod \"279b2c3c-1804-4129-9ce6-d8bcfd847096\" (UID: \"279b2c3c-1804-4129-9ce6-d8bcfd847096\") " Oct 31 19:56:29.093413 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.093379 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/279b2c3c-1804-4129-9ce6-d8bcfd847096-maas-api-openshift-ca" pod "279b2c3c-1804-4129-9ce6-d8bcfd847096" (UID: "279b2c3c-1804-4129-9ce6-d8bcfd847096"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:56:29.126934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.123107 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279b2c3c-1804-4129-9ce6-d8bcfd847096-kube-api-access-wxdsx" pod "279b2c3c-1804-4129-9ce6-d8bcfd847096" (UID: "279b2c3c-1804-4129-9ce6-d8bcfd847096"). InnerVolumeSpecName "kube-api-access-wxdsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:56:29.166970 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.166446 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/279b2c3c-1804-4129-9ce6-d8bcfd847096-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:29.166970 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.166495 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxdsx\" (UniqueName: \"kubernetes.io/projected/279b2c3c-1804-4129-9ce6-d8bcfd847096-kube-api-access-wxdsx\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:29.167275 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.167227 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/279b2c3c-1804-4129-9ce6-d8bcfd847096-maas-api-xks-ca" pod "279b2c3c-1804-4129-9ce6-d8bcfd847096" (UID: "279b2c3c-1804-4129-9ce6-d8bcfd847096"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:56:29.255467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.246335 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 31 19:56:29.255467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.246393 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" podUID="ce05955e-1925-4469-978f-29f94fc3d6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 31 19:56:29.268126 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.268097 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/279b2c3c-1804-4129-9ce6-d8bcfd847096-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:29.645983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.643115 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:56:29.765932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.764012 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm" Oct 31 19:56:29.775941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.774094 4910 generic.go:356] "Generic (PLEG): container finished" podID="c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0" containerID="aba5df7861a9f680491ef00e55f4b40eade180552490970cc19cea0a2cb3733b" exitCode=0 Oct 31 19:56:29.778951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.776134 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-workload-socket\") pod \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " Oct 31 19:56:29.778951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.776275 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-token\" (UniqueName: \"kubernetes.io/projected/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-token\") pod \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " Oct 31 19:56:29.778951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.776388 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-data\") pod \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " Oct 31 19:56:29.778951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.776427 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-workload-certs\") pod \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " Oct 31 19:56:29.778951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.776490 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-credential-socket\") pod \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " Oct 31 19:56:29.778951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.776620 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-kube-api-access-ljp8q\" (UniqueName: \"kubernetes.io/projected/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-kube-api-access-ljp8q\") pod \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " Oct 31 19:56:29.778951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.776668 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-podinfo\") pod \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " Oct 31 19:56:29.778951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.776763 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istiod-ca-cert\") pod \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " Oct 31 19:56:29.778951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.776833 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-envoy\") pod \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\" (UID: \"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08\") " Oct 31 19:56:29.778951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.777227 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-workload-certs" pod "ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08" (UID: "ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:29.778951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.777514 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-workload-socket" pod "ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08" (UID: "ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:29.778951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.777741 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-data" pod "ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08" (UID: "ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:29.779791 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.779490 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istiod-ca-cert" pod "ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08" (UID: "ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:56:29.781939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.780181 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-credential-socket" pod "ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08" (UID: "ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:29.782712 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.782678 4910 generic.go:356] "Generic (PLEG): container finished" podID="1f13c4cd-264c-44fc-814b-9dee39280ca1" containerID="8d85361589025d9bbf9c8670a78a7d8311566a9db402db386e1e07344ef11211" exitCode=0 Oct 31 19:56:29.789061 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.789034 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:29.789217 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.789067 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:29.789217 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.789083 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:29.789217 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.789099 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:29.789217 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.789112 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:29.791485 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.791413 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-token" pod "ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08" (UID: "ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:56:29.792369 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.792341 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-envoy" pod "ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08" (UID: "ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:29.793133 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.793082 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-podinfo" pod "ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08" (UID: "ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:56:29.793399 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.793378 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-kube-api-access-ljp8q" pod "ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08" (UID: "ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08"). InnerVolumeSpecName "kube-api-access-ljp8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:56:29.795277 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.795247 4910 generic.go:356] "Generic (PLEG): container finished" podID="ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08" containerID="2fbb5c4e654e2ab3c176bc4217fb13c5367740a612ab358e8658572099e62958" exitCode=0 Oct 31 19:56:29.795487 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.795468 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" Oct 31 19:56:29.817862 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.817815 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm" event={"ID":"279b2c3c-1804-4129-9ce6-d8bcfd847096","Type":"ContainerDied","Data":"b94a6b8f91508b9cde3986e1d31d8ec94c0e680315ade70fed270a7d16700336"} Oct 31 19:56:29.818063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.817873 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b94a6b8f91508b9cde3986e1d31d8ec94c0e680315ade70fed270a7d16700336" Oct 31 19:56:29.818063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.817895 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" event={"ID":"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0","Type":"ContainerDied","Data":"aba5df7861a9f680491ef00e55f4b40eade180552490970cc19cea0a2cb3733b"} Oct 31 19:56:29.818063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.817978 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" event={"ID":"1f13c4cd-264c-44fc-814b-9dee39280ca1","Type":"ContainerDied","Data":"8d85361589025d9bbf9c8670a78a7d8311566a9db402db386e1e07344ef11211"} Oct 31 19:56:29.818063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.818015 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" event={"ID":"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08","Type":"ContainerDied","Data":"2fbb5c4e654e2ab3c176bc4217fb13c5367740a612ab358e8658572099e62958"} Oct 31 19:56:29.818063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.818043 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx" event={"ID":"ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08","Type":"ContainerDied","Data":"b4fcf9d343870a8d8c4f10d75edce593112f3cf9da5f844ac1ba6358088495b0"} Oct 31 19:56:29.818353 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.818082 4910 scope.go:122] "RemoveContainer" containerID="2fbb5c4e654e2ab3c176bc4217fb13c5367740a612ab358e8658572099e62958" Oct 31 19:56:29.859935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.853016 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx"] Oct 31 19:56:29.874044 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.873897 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:56:29.877282 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.876888 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-p5bxx"] Oct 31 19:56:29.879282 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.879260 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:56:29.894877 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.894415 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljp8q\" (UniqueName: \"kubernetes.io/projected/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-kube-api-access-ljp8q\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:29.894877 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.894448 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:29.894877 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.894463 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:29.894877 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.894476 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:29.898314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.898023 4910 scope.go:122] "RemoveContainer" containerID="2fbb5c4e654e2ab3c176bc4217fb13c5367740a612ab358e8658572099e62958" Oct 31 19:56:29.898489 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:56:29.898342 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fbb5c4e654e2ab3c176bc4217fb13c5367740a612ab358e8658572099e62958\": container with ID starting with 2fbb5c4e654e2ab3c176bc4217fb13c5367740a612ab358e8658572099e62958 not found: ID does not exist" containerID="2fbb5c4e654e2ab3c176bc4217fb13c5367740a612ab358e8658572099e62958" Oct 31 19:56:29.898489 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:29.898371 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fbb5c4e654e2ab3c176bc4217fb13c5367740a612ab358e8658572099e62958"} err="failed to get container status \"2fbb5c4e654e2ab3c176bc4217fb13c5367740a612ab358e8658572099e62958\": rpc error: code = NotFound desc = could not find container \"2fbb5c4e654e2ab3c176bc4217fb13c5367740a612ab358e8658572099e62958\": container with ID starting with 2fbb5c4e654e2ab3c176bc4217fb13c5367740a612ab358e8658572099e62958 not found: ID does not exist" Oct 31 19:56:30.005602 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.005548 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-workload-certs\") pod \"1f13c4cd-264c-44fc-814b-9dee39280ca1\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " Oct 31 19:56:30.028938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.019076 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-kube-api-access-j2j4f\" (UniqueName: \"kubernetes.io/projected/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-kube-api-access-j2j4f\") pod \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " Oct 31 19:56:30.028938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.019173 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-data\") pod \"1f13c4cd-264c-44fc-814b-9dee39280ca1\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " Oct 31 19:56:30.028938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.019221 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-workload-certs\") pod \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " Oct 31 19:56:30.028938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.019344 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-credential-socket\") pod \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " Oct 31 19:56:30.028938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.019376 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-envoy\") pod \"1f13c4cd-264c-44fc-814b-9dee39280ca1\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " Oct 31 19:56:30.028938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.019432 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-workload-socket\") pod \"1f13c4cd-264c-44fc-814b-9dee39280ca1\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " Oct 31 19:56:30.028938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.019477 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-envoy\") pod \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " Oct 31 19:56:30.050938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.042086 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-workload-certs" pod "1f13c4cd-264c-44fc-814b-9dee39280ca1" (UID: "1f13c4cd-264c-44fc-814b-9dee39280ca1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:30.050938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.042225 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f13c4cd-264c-44fc-814b-9dee39280ca1-kube-api-access-9kx4b\" (UniqueName: \"kubernetes.io/projected/1f13c4cd-264c-44fc-814b-9dee39280ca1-kube-api-access-9kx4b\") pod \"1f13c4cd-264c-44fc-814b-9dee39280ca1\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.072124 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-data" pod "1f13c4cd-264c-44fc-814b-9dee39280ca1" (UID: "1f13c4cd-264c-44fc-814b-9dee39280ca1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.072516 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-workload-certs" pod "c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0" (UID: "c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.072751 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-credential-socket" pod "c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0" (UID: "c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.047144 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-podinfo\") pod \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.074655 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-token\" (UniqueName: \"kubernetes.io/projected/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-token\") pod \"1f13c4cd-264c-44fc-814b-9dee39280ca1\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.074726 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istiod-ca-cert\") pod \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.074783 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-data\") pod \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.074826 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-token\" (UniqueName: \"kubernetes.io/projected/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-token\") pod \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.074899 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-workload-socket\") pod \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\" (UID: \"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0\") " Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.074960 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-credential-socket\") pod \"1f13c4cd-264c-44fc-814b-9dee39280ca1\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.074991 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f13c4cd-264c-44fc-814b-9dee39280ca1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f13c4cd-264c-44fc-814b-9dee39280ca1-istiod-ca-cert\") pod \"1f13c4cd-264c-44fc-814b-9dee39280ca1\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.075054 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-podinfo\") pod \"1f13c4cd-264c-44fc-814b-9dee39280ca1\" (UID: \"1f13c4cd-264c-44fc-814b-9dee39280ca1\") " Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.087554 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-workload-socket" pod "1f13c4cd-264c-44fc-814b-9dee39280ca1" (UID: "1f13c4cd-264c-44fc-814b-9dee39280ca1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.087610 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-envoy" pod "1f13c4cd-264c-44fc-814b-9dee39280ca1" (UID: "1f13c4cd-264c-44fc-814b-9dee39280ca1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.089102 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-kube-api-access-j2j4f" pod "c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0" (UID: "c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0"). InnerVolumeSpecName "kube-api-access-j2j4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.089136 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-data" pod "c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0" (UID: "c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.090363 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-credential-socket" pod "1f13c4cd-264c-44fc-814b-9dee39280ca1" (UID: "1f13c4cd-264c-44fc-814b-9dee39280ca1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.090373 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.090411 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2j4f\" (UniqueName: \"kubernetes.io/projected/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-kube-api-access-j2j4f\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.090432 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.090447 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.090462 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.090478 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.090492 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.090508 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.089889 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f13c4cd-264c-44fc-814b-9dee39280ca1-istiod-ca-cert" pod "1f13c4cd-264c-44fc-814b-9dee39280ca1" (UID: "1f13c4cd-264c-44fc-814b-9dee39280ca1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.090543 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-workload-socket" pod "c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0" (UID: "c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.090626 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-envoy" pod "c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0" (UID: "c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.090634 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-podinfo" pod "c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0" (UID: "c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:56:30.092983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.090904 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istiod-ca-cert" pod "c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0" (UID: "c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:56:30.096050 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.093154 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-podinfo" pod "1f13c4cd-264c-44fc-814b-9dee39280ca1" (UID: "1f13c4cd-264c-44fc-814b-9dee39280ca1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:56:30.096050 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.093264 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-token" pod "1f13c4cd-264c-44fc-814b-9dee39280ca1" (UID: "1f13c4cd-264c-44fc-814b-9dee39280ca1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:56:30.096050 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.094545 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f13c4cd-264c-44fc-814b-9dee39280ca1-kube-api-access-9kx4b" pod "1f13c4cd-264c-44fc-814b-9dee39280ca1" (UID: "1f13c4cd-264c-44fc-814b-9dee39280ca1"). InnerVolumeSpecName "kube-api-access-9kx4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:56:30.096050 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.095115 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-token" pod "c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0" (UID: "c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:56:30.192273 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.192174 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:30.192273 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.192203 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kx4b\" (UniqueName: \"kubernetes.io/projected/1f13c4cd-264c-44fc-814b-9dee39280ca1-kube-api-access-9kx4b\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:30.192273 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.192221 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:30.192273 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.192232 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:30.192273 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.192242 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:30.192273 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.192250 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:30.192273 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.192260 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:30.192273 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.192268 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f13c4cd-264c-44fc-814b-9dee39280ca1-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:30.192273 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.192276 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f13c4cd-264c-44fc-814b-9dee39280ca1-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:30.192273 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.192285 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f13c4cd-264c-44fc-814b-9dee39280ca1-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:30.245617 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.245450 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 31 19:56:30.245617 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.245535 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" podUID="ce05955e-1925-4469-978f-29f94fc3d6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 31 19:56:30.805043 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.805012 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" Oct 31 19:56:30.805043 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.805034 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk" event={"ID":"c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0","Type":"ContainerDied","Data":"e76df673bb84f2da56360138edafd407f7015b069ce80d01a14e54d8a5ff193f"} Oct 31 19:56:30.805289 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.805095 4910 scope.go:122] "RemoveContainer" containerID="aba5df7861a9f680491ef00e55f4b40eade180552490970cc19cea0a2cb3733b" Oct 31 19:56:30.806947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.806888 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" Oct 31 19:56:30.807081 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.806890 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf" event={"ID":"1f13c4cd-264c-44fc-814b-9dee39280ca1","Type":"ContainerDied","Data":"5c130117c6bc3b95212d7d84cb01dfaf33954c74ba95753c790f2e8081ae5a2c"} Oct 31 19:56:30.817830 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.817814 4910 scope.go:122] "RemoveContainer" containerID="8d85361589025d9bbf9c8670a78a7d8311566a9db402db386e1e07344ef11211" Oct 31 19:56:30.858338 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.858307 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk"] Oct 31 19:56:30.866858 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.866832 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-q4mvk"] Oct 31 19:56:30.938937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.938769 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf"] Oct 31 19:56:30.985413 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:30.983390 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-ktzsf"] Oct 31 19:56:31.001356 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:31.001326 4910 scope.go:122] "RemoveContainer" containerID="f5b4e67fbc6726b73cba2ac470afc154cb73562f8ae109a5623d0f9a04c9c8cf" Oct 31 19:56:31.012398 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:31.012376 4910 scope.go:122] "RemoveContainer" containerID="6feffa55d7f772b8968711118dd7252c8bb548c8f48218199a9f07945e64acce" Oct 31 19:56:31.245091 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:31.245056 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 31 19:56:31.245251 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:31.245111 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" podUID="ce05955e-1925-4469-978f-29f94fc3d6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 31 19:56:31.760669 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:31.760635 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f13c4cd-264c-44fc-814b-9dee39280ca1" path="/var/lib/kubelet/pods/1f13c4cd-264c-44fc-814b-9dee39280ca1/volumes" Oct 31 19:56:31.761327 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:31.761309 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08" path="/var/lib/kubelet/pods/ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08/volumes" Oct 31 19:56:31.761938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:31.761900 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0" path="/var/lib/kubelet/pods/c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0/volumes" Oct 31 19:56:32.250966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:32.250934 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:32.251391 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:32.251255 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:32.252130 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:32.252112 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 19:56:34.992727 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:34.992688 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv"] Oct 31 19:56:38.973932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:38.969343 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm"] Oct 31 19:56:39.023021 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:39.018892 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7c892a-jv6hm"] Oct 31 19:56:39.759394 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:39.759359 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="279b2c3c-1804-4129-9ce6-d8bcfd847096" path="/var/lib/kubelet/pods/279b2c3c-1804-4129-9ce6-d8bcfd847096/volumes" Oct 31 19:56:39.994815 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:39.994757 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv" podUID="63a3c457-7089-49bf-8ed5-6b18b933b832" containerName="payload-processing" containerID="cri-o://bb817e5db3213ca1dbf220f8be639e9b103be94bd004d5f5013adedf7f3907c6" gracePeriod=25 Oct 31 19:56:40.311697 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:40.311669 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv" Oct 31 19:56:40.334091 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:40.334062 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63a3c457-7089-49bf-8ed5-6b18b933b832-kube-api-access-cfz6r\" (UniqueName: \"kubernetes.io/projected/63a3c457-7089-49bf-8ed5-6b18b933b832-kube-api-access-cfz6r\") pod \"63a3c457-7089-49bf-8ed5-6b18b933b832\" (UID: \"63a3c457-7089-49bf-8ed5-6b18b933b832\") " Oct 31 19:56:40.334355 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:40.334339 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63a3c457-7089-49bf-8ed5-6b18b933b832-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a3c457-7089-49bf-8ed5-6b18b933b832-plugins-config-volume\") pod \"63a3c457-7089-49bf-8ed5-6b18b933b832\" (UID: \"63a3c457-7089-49bf-8ed5-6b18b933b832\") " Oct 31 19:56:40.334712 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:40.334686 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63a3c457-7089-49bf-8ed5-6b18b933b832-plugins-config-volume" pod "63a3c457-7089-49bf-8ed5-6b18b933b832" (UID: "63a3c457-7089-49bf-8ed5-6b18b933b832"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:56:40.335117 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:40.335098 4910 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63a3c457-7089-49bf-8ed5-6b18b933b832-plugins-config-volume\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:40.355215 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:40.354542 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a3c457-7089-49bf-8ed5-6b18b933b832-kube-api-access-cfz6r" pod "63a3c457-7089-49bf-8ed5-6b18b933b832" (UID: "63a3c457-7089-49bf-8ed5-6b18b933b832"). InnerVolumeSpecName "kube-api-access-cfz6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:56:40.436537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:40.436501 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfz6r\" (UniqueName: \"kubernetes.io/projected/63a3c457-7089-49bf-8ed5-6b18b933b832-kube-api-access-cfz6r\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:41.042996 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:41.042957 4910 generic.go:356] "Generic (PLEG): container finished" podID="63a3c457-7089-49bf-8ed5-6b18b933b832" containerID="bb817e5db3213ca1dbf220f8be639e9b103be94bd004d5f5013adedf7f3907c6" exitCode=0 Oct 31 19:56:41.043556 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:41.043048 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv" Oct 31 19:56:41.043556 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:41.043056 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv" event={"ID":"63a3c457-7089-49bf-8ed5-6b18b933b832","Type":"ContainerDied","Data":"bb817e5db3213ca1dbf220f8be639e9b103be94bd004d5f5013adedf7f3907c6"} Oct 31 19:56:41.043556 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:41.043127 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv" event={"ID":"63a3c457-7089-49bf-8ed5-6b18b933b832","Type":"ContainerDied","Data":"2ab27a890cc2f991e334ed10d5a0e8d89b354461ffb7f564ea4a555eacedf4bd"} Oct 31 19:56:41.043556 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:41.043161 4910 scope.go:122] "RemoveContainer" containerID="bb817e5db3213ca1dbf220f8be639e9b103be94bd004d5f5013adedf7f3907c6" Oct 31 19:56:41.063228 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:41.063203 4910 scope.go:122] "RemoveContainer" containerID="bb817e5db3213ca1dbf220f8be639e9b103be94bd004d5f5013adedf7f3907c6" Oct 31 19:56:41.063575 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:56:41.063553 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb817e5db3213ca1dbf220f8be639e9b103be94bd004d5f5013adedf7f3907c6\": container with ID starting with bb817e5db3213ca1dbf220f8be639e9b103be94bd004d5f5013adedf7f3907c6 not found: ID does not exist" containerID="bb817e5db3213ca1dbf220f8be639e9b103be94bd004d5f5013adedf7f3907c6" Oct 31 19:56:41.063651 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:41.063583 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb817e5db3213ca1dbf220f8be639e9b103be94bd004d5f5013adedf7f3907c6"} err="failed to get container status \"bb817e5db3213ca1dbf220f8be639e9b103be94bd004d5f5013adedf7f3907c6\": rpc error: code = NotFound desc = could not find container \"bb817e5db3213ca1dbf220f8be639e9b103be94bd004d5f5013adedf7f3907c6\": container with ID starting with bb817e5db3213ca1dbf220f8be639e9b103be94bd004d5f5013adedf7f3907c6 not found: ID does not exist" Oct 31 19:56:41.074519 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:41.074492 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv"] Oct 31 19:56:41.081601 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:41.081575 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7c892a-58845567f5-kgksv"] Oct 31 19:56:41.759190 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:41.759159 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63a3c457-7089-49bf-8ed5-6b18b933b832" path="/var/lib/kubelet/pods/63a3c457-7089-49bf-8ed5-6b18b933b832/volumes" Oct 31 19:56:42.716848 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:56:42.716811 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:56:47.246270 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:47.246234 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9"] Oct 31 19:56:47.246724 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:47.246499 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" podUID="7c03ae04-2fbb-44b2-ac0d-25c39f35608b" containerName="istio-proxy" containerID="cri-o://6f10b490f9dc5d9246a2c39544d0dd923911d9a5c5949bca71b21e918fbe8ef9" gracePeriod=30 Oct 31 19:56:47.404199 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:47.404136 4910 patch_prober.go:29] interesting pod/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:56:47.404523 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:47.404478 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" podUID="7c03ae04-2fbb-44b2-ac0d-25c39f35608b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:56:51.572385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:51.572339 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc"] Oct 31 19:56:51.573590 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:51.573557 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" podUID="9bff40e4-ffc5-4104-8aed-4ff06167396c" containerName="istio-proxy" containerID="cri-o://22b01f8018ba6948f24c94d48d93246192f7e9fcd4f1aeb397b1994908b5d23f" gracePeriod=30 Oct 31 19:56:51.775381 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:51.775347 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr"] Oct 31 19:56:51.775601 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:51.775576 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" podUID="16761f60-662e-4eb1-a247-b00378149cc4" containerName="istio-proxy" containerID="cri-o://522881406a390a05402383a2d7dba0ae41adfe62527712ca787e5c2ecc06a5c5" gracePeriod=30 Oct 31 19:56:51.870832 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:51.870742 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48"] Oct 31 19:56:51.871044 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:51.870950 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" podUID="cc8a6def-fe67-42b8-aea6-b3cd2399def1" containerName="istio-proxy" containerID="cri-o://f3b13311a7417606d2db084c1931e25fa6191b2867223543b624a4494dd0138d" gracePeriod=30 Oct 31 19:56:52.625151 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.625113 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:56:52.694383 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.694340 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-podinfo\") pod \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " Oct 31 19:56:52.694591 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.694399 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istiod-ca-cert\") pod \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " Oct 31 19:56:52.694591 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.694495 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-kube-api-access-vj86c\" (UniqueName: \"kubernetes.io/projected/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-kube-api-access-vj86c\") pod \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " Oct 31 19:56:52.694591 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.694545 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-envoy\") pod \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " Oct 31 19:56:52.694767 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.694679 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-workload-socket\") pod \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " Oct 31 19:56:52.694869 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.694848 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-credential-socket\") pod \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " Oct 31 19:56:52.694948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.694899 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-token\" (UniqueName: \"kubernetes.io/projected/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-token\") pod \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " Oct 31 19:56:52.694948 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.694943 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-workload-certs\") pod \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " Oct 31 19:56:52.695074 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.695025 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-data\") pod \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\" (UID: \"7c03ae04-2fbb-44b2-ac0d-25c39f35608b\") " Oct 31 19:56:52.695354 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.695320 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-credential-socket" pod "7c03ae04-2fbb-44b2-ac0d-25c39f35608b" (UID: "7c03ae04-2fbb-44b2-ac0d-25c39f35608b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:52.695621 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.695595 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-data" pod "7c03ae04-2fbb-44b2-ac0d-25c39f35608b" (UID: "7c03ae04-2fbb-44b2-ac0d-25c39f35608b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:52.696035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.696011 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:52.696143 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.696040 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:52.696263 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.696209 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-workload-socket" pod "7c03ae04-2fbb-44b2-ac0d-25c39f35608b" (UID: "7c03ae04-2fbb-44b2-ac0d-25c39f35608b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:52.696338 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.696298 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-workload-certs" pod "7c03ae04-2fbb-44b2-ac0d-25c39f35608b" (UID: "7c03ae04-2fbb-44b2-ac0d-25c39f35608b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:52.699719 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.699690 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-token" pod "7c03ae04-2fbb-44b2-ac0d-25c39f35608b" (UID: "7c03ae04-2fbb-44b2-ac0d-25c39f35608b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:56:52.700114 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.700086 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istiod-ca-cert" pod "7c03ae04-2fbb-44b2-ac0d-25c39f35608b" (UID: "7c03ae04-2fbb-44b2-ac0d-25c39f35608b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:56:52.704437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.704400 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-podinfo" pod "7c03ae04-2fbb-44b2-ac0d-25c39f35608b" (UID: "7c03ae04-2fbb-44b2-ac0d-25c39f35608b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:56:52.704674 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.704645 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-kube-api-access-vj86c" pod "7c03ae04-2fbb-44b2-ac0d-25c39f35608b" (UID: "7c03ae04-2fbb-44b2-ac0d-25c39f35608b"). InnerVolumeSpecName "kube-api-access-vj86c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:56:52.706601 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.706567 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-envoy" pod "7c03ae04-2fbb-44b2-ac0d-25c39f35608b" (UID: "7c03ae04-2fbb-44b2-ac0d-25c39f35608b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:52.797618 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.797581 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:52.797618 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.797610 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:52.797618 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.797624 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj86c\" (UniqueName: \"kubernetes.io/projected/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-kube-api-access-vj86c\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:52.797861 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.797637 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:52.797861 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.797648 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:52.797861 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.797660 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:52.797861 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:52.797671 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c03ae04-2fbb-44b2-ac0d-25c39f35608b-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:53.171754 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:53.171716 4910 generic.go:356] "Generic (PLEG): container finished" podID="7c03ae04-2fbb-44b2-ac0d-25c39f35608b" containerID="6f10b490f9dc5d9246a2c39544d0dd923911d9a5c5949bca71b21e918fbe8ef9" exitCode=0 Oct 31 19:56:53.172008 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:53.171786 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" Oct 31 19:56:53.172008 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:53.171803 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" event={"ID":"7c03ae04-2fbb-44b2-ac0d-25c39f35608b","Type":"ContainerDied","Data":"6f10b490f9dc5d9246a2c39544d0dd923911d9a5c5949bca71b21e918fbe8ef9"} Oct 31 19:56:53.172008 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:53.171941 4910 scope.go:122] "RemoveContainer" containerID="6f10b490f9dc5d9246a2c39544d0dd923911d9a5c5949bca71b21e918fbe8ef9" Oct 31 19:56:53.172209 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:53.171950 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9" event={"ID":"7c03ae04-2fbb-44b2-ac0d-25c39f35608b","Type":"ContainerDied","Data":"119f5383195ab64f44711c7a6e3bd272c7a30df7e0a9582d5106bcc479f2fb88"} Oct 31 19:56:53.185108 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:53.185088 4910 scope.go:122] "RemoveContainer" containerID="4af5eafc113132b8b8d8692f3602fae614f5c32cea538c31eee71d45e9dd83d6" Oct 31 19:56:53.196547 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:53.196522 4910 scope.go:122] "RemoveContainer" containerID="6f10b490f9dc5d9246a2c39544d0dd923911d9a5c5949bca71b21e918fbe8ef9" Oct 31 19:56:53.196855 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:56:53.196833 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f10b490f9dc5d9246a2c39544d0dd923911d9a5c5949bca71b21e918fbe8ef9\": container with ID starting with 6f10b490f9dc5d9246a2c39544d0dd923911d9a5c5949bca71b21e918fbe8ef9 not found: ID does not exist" containerID="6f10b490f9dc5d9246a2c39544d0dd923911d9a5c5949bca71b21e918fbe8ef9" Oct 31 19:56:53.196981 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:53.196864 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f10b490f9dc5d9246a2c39544d0dd923911d9a5c5949bca71b21e918fbe8ef9"} err="failed to get container status \"6f10b490f9dc5d9246a2c39544d0dd923911d9a5c5949bca71b21e918fbe8ef9\": rpc error: code = NotFound desc = could not find container \"6f10b490f9dc5d9246a2c39544d0dd923911d9a5c5949bca71b21e918fbe8ef9\": container with ID starting with 6f10b490f9dc5d9246a2c39544d0dd923911d9a5c5949bca71b21e918fbe8ef9 not found: ID does not exist" Oct 31 19:56:53.196981 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:53.196881 4910 scope.go:122] "RemoveContainer" containerID="4af5eafc113132b8b8d8692f3602fae614f5c32cea538c31eee71d45e9dd83d6" Oct 31 19:56:53.197393 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:56:53.197358 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4af5eafc113132b8b8d8692f3602fae614f5c32cea538c31eee71d45e9dd83d6\": container with ID starting with 4af5eafc113132b8b8d8692f3602fae614f5c32cea538c31eee71d45e9dd83d6 not found: ID does not exist" containerID="4af5eafc113132b8b8d8692f3602fae614f5c32cea538c31eee71d45e9dd83d6" Oct 31 19:56:53.197478 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:53.197403 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4af5eafc113132b8b8d8692f3602fae614f5c32cea538c31eee71d45e9dd83d6"} err="failed to get container status \"4af5eafc113132b8b8d8692f3602fae614f5c32cea538c31eee71d45e9dd83d6\": rpc error: code = NotFound desc = could not find container \"4af5eafc113132b8b8d8692f3602fae614f5c32cea538c31eee71d45e9dd83d6\": container with ID starting with 4af5eafc113132b8b8d8692f3602fae614f5c32cea538c31eee71d45e9dd83d6 not found: ID does not exist" Oct 31 19:56:53.202665 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:53.202643 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9"] Oct 31 19:56:53.208871 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:53.208849 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-pr4j9"] Oct 31 19:56:53.758485 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:53.758454 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c03ae04-2fbb-44b2-ac0d-25c39f35608b" path="/var/lib/kubelet/pods/7c03ae04-2fbb-44b2-ac0d-25c39f35608b/volumes" Oct 31 19:56:54.000075 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.000037 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp"] Oct 31 19:56:54.000789 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.000765 4910 cpu_manager.go:414] "containerMap: removing container" podUID="1f13c4cd-264c-44fc-814b-9dee39280ca1" containerName="istio-proxy" Oct 31 19:56:54.000789 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.000788 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f13c4cd-264c-44fc-814b-9dee39280ca1" containerName="istio-proxy" Oct 31 19:56:54.001036 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.000820 4910 cpu_manager.go:414] "containerMap: removing container" podUID="ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08" containerName="istio-proxy" Oct 31 19:56:54.001036 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.000829 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08" containerName="istio-proxy" Oct 31 19:56:54.001036 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.000848 4910 cpu_manager.go:414] "containerMap: removing container" podUID="7c03ae04-2fbb-44b2-ac0d-25c39f35608b" containerName="istio-proxy" Oct 31 19:56:54.001036 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.000856 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c03ae04-2fbb-44b2-ac0d-25c39f35608b" containerName="istio-proxy" Oct 31 19:56:54.001036 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.000876 4910 cpu_manager.go:414] "containerMap: removing container" podUID="7c03ae04-2fbb-44b2-ac0d-25c39f35608b" containerName="istio-proxy" Oct 31 19:56:54.001036 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.000882 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c03ae04-2fbb-44b2-ac0d-25c39f35608b" containerName="istio-proxy" Oct 31 19:56:54.001036 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.000906 4910 cpu_manager.go:414] "containerMap: removing container" podUID="279b2c3c-1804-4129-9ce6-d8bcfd847096" containerName="revoke-keys" Oct 31 19:56:54.001036 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.000929 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="279b2c3c-1804-4129-9ce6-d8bcfd847096" containerName="revoke-keys" Oct 31 19:56:54.001036 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.000973 4910 cpu_manager.go:414] "containerMap: removing container" podUID="63a3c457-7089-49bf-8ed5-6b18b933b832" containerName="payload-processing" Oct 31 19:56:54.001036 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.000980 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63a3c457-7089-49bf-8ed5-6b18b933b832" containerName="payload-processing" Oct 31 19:56:54.001036 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.000992 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0" containerName="istio-proxy" Oct 31 19:56:54.001036 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.001000 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0" containerName="istio-proxy" Oct 31 19:56:54.001635 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.001180 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="63a3c457-7089-49bf-8ed5-6b18b933b832" containerName="payload-processing" Oct 31 19:56:54.001635 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.001197 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4eb3a4c-2d97-4695-b5c4-ca36d2d75da0" containerName="istio-proxy" Oct 31 19:56:54.001635 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.001208 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f13c4cd-264c-44fc-814b-9dee39280ca1" containerName="istio-proxy" Oct 31 19:56:54.001635 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.001221 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba7a7db9-37d7-4e4e-9126-3ff5b1c91d08" containerName="istio-proxy" Oct 31 19:56:54.001635 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.001232 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c03ae04-2fbb-44b2-ac0d-25c39f35608b" containerName="istio-proxy" Oct 31 19:56:54.001635 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.001244 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="279b2c3c-1804-4129-9ce6-d8bcfd847096" containerName="revoke-keys" Oct 31 19:56:54.006543 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.006520 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp" Oct 31 19:56:54.015698 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.015675 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1f2781-dockercfg-8zcsk\"" Oct 31 19:56:54.017308 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.017274 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1f2781\"" Oct 31 19:56:54.023403 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.023373 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg"] Oct 31 19:56:54.024569 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.024545 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c03ae04-2fbb-44b2-ac0d-25c39f35608b" containerName="istio-proxy" Oct 31 19:56:54.042968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.042324 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp"] Oct 31 19:56:54.042968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.042388 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg"] Oct 31 19:56:54.042968 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.042401 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg" Oct 31 19:56:54.179940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.172150 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mflpb\" (UniqueName: \"kubernetes.io/projected/2f0e2560-f3d7-4ccc-b139-3f8423f667a1-kube-api-access-mflpb\") pod \"payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp\" (UID: \"2f0e2560-f3d7-4ccc-b139-3f8423f667a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp" Oct 31 19:56:54.179940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.172282 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3572ce0-4758-4745-8312-3b9bcc942762-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg\" (UID: \"d3572ce0-4758-4745-8312-3b9bcc942762\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg" Oct 31 19:56:54.179940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.172476 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dznn\" (UniqueName: \"kubernetes.io/projected/d3572ce0-4758-4745-8312-3b9bcc942762-kube-api-access-4dznn\") pod \"payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg\" (UID: \"d3572ce0-4758-4745-8312-3b9bcc942762\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg" Oct 31 19:56:54.179940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.172546 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0e2560-f3d7-4ccc-b139-3f8423f667a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp\" (UID: \"2f0e2560-f3d7-4ccc-b139-3f8423f667a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp" Oct 31 19:56:54.276989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.276780 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dznn\" (UniqueName: \"kubernetes.io/projected/d3572ce0-4758-4745-8312-3b9bcc942762-kube-api-access-4dznn\") pod \"payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg\" (UID: \"d3572ce0-4758-4745-8312-3b9bcc942762\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg" Oct 31 19:56:54.276989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.276872 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0e2560-f3d7-4ccc-b139-3f8423f667a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp\" (UID: \"2f0e2560-f3d7-4ccc-b139-3f8423f667a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp" Oct 31 19:56:54.277258 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.277049 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mflpb\" (UniqueName: \"kubernetes.io/projected/2f0e2560-f3d7-4ccc-b139-3f8423f667a1-kube-api-access-mflpb\") pod \"payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp\" (UID: \"2f0e2560-f3d7-4ccc-b139-3f8423f667a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp" Oct 31 19:56:54.277258 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.277140 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3572ce0-4758-4745-8312-3b9bcc942762-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg\" (UID: \"d3572ce0-4758-4745-8312-3b9bcc942762\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg" Oct 31 19:56:54.285505 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.278069 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3572ce0-4758-4745-8312-3b9bcc942762-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg\" (UID: \"d3572ce0-4758-4745-8312-3b9bcc942762\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg" Oct 31 19:56:54.285505 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.279237 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0e2560-f3d7-4ccc-b139-3f8423f667a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp\" (UID: \"2f0e2560-f3d7-4ccc-b139-3f8423f667a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp" Oct 31 19:56:54.316939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.316461 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dznn\" (UniqueName: \"kubernetes.io/projected/d3572ce0-4758-4745-8312-3b9bcc942762-kube-api-access-4dznn\") pod \"payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg\" (UID: \"d3572ce0-4758-4745-8312-3b9bcc942762\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg" Oct 31 19:56:54.324934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.319312 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mflpb\" (UniqueName: \"kubernetes.io/projected/2f0e2560-f3d7-4ccc-b139-3f8423f667a1-kube-api-access-mflpb\") pod \"payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp\" (UID: \"2f0e2560-f3d7-4ccc-b139-3f8423f667a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp" Oct 31 19:56:54.332934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.332329 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp" Oct 31 19:56:54.364594 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.364549 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg" Oct 31 19:56:54.733244 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:56:54.733203 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3572ce0_4758_4745_8312_3b9bcc942762.slice/crio-81504bac03d08fbd5cce4a45ac72da78dcf45775e3d0fdc397f34ea2d5294da0 WatchSource:0}: Error finding container 81504bac03d08fbd5cce4a45ac72da78dcf45775e3d0fdc397f34ea2d5294da0: Status 404 returned error can't find the container with id 81504bac03d08fbd5cce4a45ac72da78dcf45775e3d0fdc397f34ea2d5294da0 Oct 31 19:56:54.734276 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.734251 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg"] Oct 31 19:56:54.743946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:54.743919 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp"] Oct 31 19:56:54.747819 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:56:54.747794 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0e2560_f3d7_4ccc_b139_3f8423f667a1.slice/crio-550fdc8c861ac91ce34e84e4db01a7e2232256c02cd8e30efd6867200dc7fbba WatchSource:0}: Error finding container 550fdc8c861ac91ce34e84e4db01a7e2232256c02cd8e30efd6867200dc7fbba: Status 404 returned error can't find the container with id 550fdc8c861ac91ce34e84e4db01a7e2232256c02cd8e30efd6867200dc7fbba Oct 31 19:56:55.033571 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:55.033477 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8772c052-zplbh"] Oct 31 19:56:55.047679 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:55.047645 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8772c052-zplbh"] Oct 31 19:56:55.254063 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:55.254025 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg" event={"ID":"d3572ce0-4758-4745-8312-3b9bcc942762","Type":"ContainerStarted","Data":"334b1ef01583f54760836d44f07664013e7382f27b24db697fde8ec61a506005"} Oct 31 19:56:55.254245 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:55.254117 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg" event={"ID":"d3572ce0-4758-4745-8312-3b9bcc942762","Type":"ContainerStarted","Data":"81504bac03d08fbd5cce4a45ac72da78dcf45775e3d0fdc397f34ea2d5294da0"} Oct 31 19:56:55.254341 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:55.254244 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg" Oct 31 19:56:55.256009 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:55.255977 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp" event={"ID":"2f0e2560-f3d7-4ccc-b139-3f8423f667a1","Type":"ContainerStarted","Data":"78a8d565f89faa69d7db046237af16d2a053c4907da51db36e9c4b6654d55db7"} Oct 31 19:56:55.256131 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:55.256027 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp" event={"ID":"2f0e2560-f3d7-4ccc-b139-3f8423f667a1","Type":"ContainerStarted","Data":"550fdc8c861ac91ce34e84e4db01a7e2232256c02cd8e30efd6867200dc7fbba"} Oct 31 19:56:55.256131 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:55.256057 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp" Oct 31 19:56:55.278265 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:55.278211 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg" podStartSLOduration=2.278198006 podStartE2EDuration="2.278198006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:56:55.276271214 +0000 UTC m=+2249.639091587" watchObservedRunningTime="2026-10-31 19:56:55.278198006 +0000 UTC m=+2249.641018378" Oct 31 19:56:55.298976 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:55.298862 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp" podStartSLOduration=2.298847228 podStartE2EDuration="2.298847228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:56:55.294365166 +0000 UTC m=+2249.657185538" watchObservedRunningTime="2026-10-31 19:56:55.298847228 +0000 UTC m=+2249.661667599" Oct 31 19:56:55.757153 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:55.757119 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbe6143b-0623-4419-a8eb-5a5047ae149a" path="/var/lib/kubelet/pods/cbe6143b-0623-4419-a8eb-5a5047ae149a/volumes" Oct 31 19:56:55.845528 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:55.845490 4910 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:56:55.845787 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:55.845760 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" podUID="9bff40e4-ffc5-4104-8aed-4ff06167396c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:56:56.239983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:56.239897 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 19:56:56.240411 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:56.240002 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" podUID="cc8a6def-fe67-42b8-aea6-b3cd2399def1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 19:56:56.993187 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:56.993105 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:56:57.113069 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.113026 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-token\" (UniqueName: \"kubernetes.io/projected/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-token\") pod \"9bff40e4-ffc5-4104-8aed-4ff06167396c\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " Oct 31 19:56:57.113263 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.113209 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-envoy\") pod \"9bff40e4-ffc5-4104-8aed-4ff06167396c\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " Oct 31 19:56:57.113263 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.113251 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-credential-socket\") pod \"9bff40e4-ffc5-4104-8aed-4ff06167396c\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " Oct 31 19:56:57.113392 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.113321 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bff40e4-ffc5-4104-8aed-4ff06167396c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bff40e4-ffc5-4104-8aed-4ff06167396c-istiod-ca-cert\") pod \"9bff40e4-ffc5-4104-8aed-4ff06167396c\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " Oct 31 19:56:57.113392 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.113360 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-podinfo\") pod \"9bff40e4-ffc5-4104-8aed-4ff06167396c\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " Oct 31 19:56:57.113507 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.113407 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bff40e4-ffc5-4104-8aed-4ff06167396c-kube-api-access-f2k9w\" (UniqueName: \"kubernetes.io/projected/9bff40e4-ffc5-4104-8aed-4ff06167396c-kube-api-access-f2k9w\") pod \"9bff40e4-ffc5-4104-8aed-4ff06167396c\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " Oct 31 19:56:57.113507 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.113479 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-workload-socket\") pod \"9bff40e4-ffc5-4104-8aed-4ff06167396c\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " Oct 31 19:56:57.113614 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.113550 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-workload-certs\") pod \"9bff40e4-ffc5-4104-8aed-4ff06167396c\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " Oct 31 19:56:57.113614 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.113606 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-data\") pod \"9bff40e4-ffc5-4104-8aed-4ff06167396c\" (UID: \"9bff40e4-ffc5-4104-8aed-4ff06167396c\") " Oct 31 19:56:57.115207 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.114863 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bff40e4-ffc5-4104-8aed-4ff06167396c-istiod-ca-cert" pod "9bff40e4-ffc5-4104-8aed-4ff06167396c" (UID: "9bff40e4-ffc5-4104-8aed-4ff06167396c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:56:57.115543 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.115496 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-data" pod "9bff40e4-ffc5-4104-8aed-4ff06167396c" (UID: "9bff40e4-ffc5-4104-8aed-4ff06167396c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:57.117119 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.117018 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-credential-socket" pod "9bff40e4-ffc5-4104-8aed-4ff06167396c" (UID: "9bff40e4-ffc5-4104-8aed-4ff06167396c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:57.117119 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.117092 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-workload-certs" pod "9bff40e4-ffc5-4104-8aed-4ff06167396c" (UID: "9bff40e4-ffc5-4104-8aed-4ff06167396c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:57.117590 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.117568 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-workload-socket" pod "9bff40e4-ffc5-4104-8aed-4ff06167396c" (UID: "9bff40e4-ffc5-4104-8aed-4ff06167396c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:57.119624 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.118531 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-podinfo" pod "9bff40e4-ffc5-4104-8aed-4ff06167396c" (UID: "9bff40e4-ffc5-4104-8aed-4ff06167396c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:56:57.130870 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.130809 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bff40e4-ffc5-4104-8aed-4ff06167396c-kube-api-access-f2k9w" pod "9bff40e4-ffc5-4104-8aed-4ff06167396c" (UID: "9bff40e4-ffc5-4104-8aed-4ff06167396c"). InnerVolumeSpecName "kube-api-access-f2k9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:56:57.132650 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.132605 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-token" pod "9bff40e4-ffc5-4104-8aed-4ff06167396c" (UID: "9bff40e4-ffc5-4104-8aed-4ff06167396c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:56:57.146994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.146881 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-envoy" pod "9bff40e4-ffc5-4104-8aed-4ff06167396c" (UID: "9bff40e4-ffc5-4104-8aed-4ff06167396c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:57.167977 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.167950 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:56:57.216130 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.216089 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bff40e4-ffc5-4104-8aed-4ff06167396c-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.216130 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.216130 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.216393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.216145 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2k9w\" (UniqueName: \"kubernetes.io/projected/9bff40e4-ffc5-4104-8aed-4ff06167396c-kube-api-access-f2k9w\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.216393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.216159 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.216393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.216173 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.216393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.216186 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.216393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.216199 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.216393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.216224 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.216393 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.216237 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bff40e4-ffc5-4104-8aed-4ff06167396c-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.255031 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.255002 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:56:57.280314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.280277 4910 generic.go:356] "Generic (PLEG): container finished" podID="cc8a6def-fe67-42b8-aea6-b3cd2399def1" containerID="f3b13311a7417606d2db084c1931e25fa6191b2867223543b624a4494dd0138d" exitCode=0 Oct 31 19:56:57.281539 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.280423 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" event={"ID":"cc8a6def-fe67-42b8-aea6-b3cd2399def1","Type":"ContainerDied","Data":"f3b13311a7417606d2db084c1931e25fa6191b2867223543b624a4494dd0138d"} Oct 31 19:56:57.281539 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.280475 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" event={"ID":"cc8a6def-fe67-42b8-aea6-b3cd2399def1","Type":"ContainerDied","Data":"f29a8bd6c9a80ccaae949596109d7148d9316d0bdcd8e9b796005cd9ca319de4"} Oct 31 19:56:57.281539 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.280504 4910 scope.go:122] "RemoveContainer" containerID="f3b13311a7417606d2db084c1931e25fa6191b2867223543b624a4494dd0138d" Oct 31 19:56:57.281539 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.280649 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48" Oct 31 19:56:57.292947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.284229 4910 generic.go:356] "Generic (PLEG): container finished" podID="16761f60-662e-4eb1-a247-b00378149cc4" containerID="522881406a390a05402383a2d7dba0ae41adfe62527712ca787e5c2ecc06a5c5" exitCode=0 Oct 31 19:56:57.292947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.284436 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" event={"ID":"16761f60-662e-4eb1-a247-b00378149cc4","Type":"ContainerDied","Data":"522881406a390a05402383a2d7dba0ae41adfe62527712ca787e5c2ecc06a5c5"} Oct 31 19:56:57.292947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.284487 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" event={"ID":"16761f60-662e-4eb1-a247-b00378149cc4","Type":"ContainerDied","Data":"d98f170fdb5c2e31b2005bfa535851c9f33cef65579b3bf69cc76f9a1df047d0"} Oct 31 19:56:57.292947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.284565 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr" Oct 31 19:56:57.292947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.286796 4910 generic.go:356] "Generic (PLEG): container finished" podID="9bff40e4-ffc5-4104-8aed-4ff06167396c" containerID="22b01f8018ba6948f24c94d48d93246192f7e9fcd4f1aeb397b1994908b5d23f" exitCode=0 Oct 31 19:56:57.292947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.286896 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" event={"ID":"9bff40e4-ffc5-4104-8aed-4ff06167396c","Type":"ContainerDied","Data":"22b01f8018ba6948f24c94d48d93246192f7e9fcd4f1aeb397b1994908b5d23f"} Oct 31 19:56:57.292947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.286977 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" event={"ID":"9bff40e4-ffc5-4104-8aed-4ff06167396c","Type":"ContainerDied","Data":"615c4a375a89f426650624cb7f4629a6f0321a90f2d17222bc109ed0415fcb6b"} Oct 31 19:56:57.292947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.287049 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc" Oct 31 19:56:57.303066 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.302972 4910 scope.go:122] "RemoveContainer" containerID="f3b13311a7417606d2db084c1931e25fa6191b2867223543b624a4494dd0138d" Oct 31 19:56:57.303353 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:56:57.303328 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3b13311a7417606d2db084c1931e25fa6191b2867223543b624a4494dd0138d\": container with ID starting with f3b13311a7417606d2db084c1931e25fa6191b2867223543b624a4494dd0138d not found: ID does not exist" containerID="f3b13311a7417606d2db084c1931e25fa6191b2867223543b624a4494dd0138d" Oct 31 19:56:57.303424 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.303364 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3b13311a7417606d2db084c1931e25fa6191b2867223543b624a4494dd0138d"} err="failed to get container status \"f3b13311a7417606d2db084c1931e25fa6191b2867223543b624a4494dd0138d\": rpc error: code = NotFound desc = could not find container \"f3b13311a7417606d2db084c1931e25fa6191b2867223543b624a4494dd0138d\": container with ID starting with f3b13311a7417606d2db084c1931e25fa6191b2867223543b624a4494dd0138d not found: ID does not exist" Oct 31 19:56:57.303424 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.303388 4910 scope.go:122] "RemoveContainer" containerID="522881406a390a05402383a2d7dba0ae41adfe62527712ca787e5c2ecc06a5c5" Oct 31 19:56:57.317188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.316679 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-workload-socket\") pod \"16761f60-662e-4eb1-a247-b00378149cc4\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " Oct 31 19:56:57.317188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.316725 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-istio-data\") pod \"16761f60-662e-4eb1-a247-b00378149cc4\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " Oct 31 19:56:57.317188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.316771 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-token\" (UniqueName: \"kubernetes.io/projected/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-token\") pod \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " Oct 31 19:56:57.317188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.316820 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16761f60-662e-4eb1-a247-b00378149cc4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16761f60-662e-4eb1-a247-b00378149cc4-istiod-ca-cert\") pod \"16761f60-662e-4eb1-a247-b00378149cc4\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " Oct 31 19:56:57.317188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.316898 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8a6def-fe67-42b8-aea6-b3cd2399def1-kube-api-access-dz6ts\" (UniqueName: \"kubernetes.io/projected/cc8a6def-fe67-42b8-aea6-b3cd2399def1-kube-api-access-dz6ts\") pod \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " Oct 31 19:56:57.317188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.316992 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-data\") pod \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " Oct 31 19:56:57.317188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.316993 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-workload-socket" pod "16761f60-662e-4eb1-a247-b00378149cc4" (UID: "16761f60-662e-4eb1-a247-b00378149cc4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:57.317627 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.317307 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16761f60-662e-4eb1-a247-b00378149cc4-istiod-ca-cert" pod "16761f60-662e-4eb1-a247-b00378149cc4" (UID: "16761f60-662e-4eb1-a247-b00378149cc4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:56:57.317627 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.317352 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istiod-ca-cert\") pod \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " Oct 31 19:56:57.317627 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.317405 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16761f60-662e-4eb1-a247-b00378149cc4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16761f60-662e-4eb1-a247-b00378149cc4-istio-podinfo\") pod \"16761f60-662e-4eb1-a247-b00378149cc4\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " Oct 31 19:56:57.317627 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.317454 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-podinfo\") pod \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " Oct 31 19:56:57.317627 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.317516 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16761f60-662e-4eb1-a247-b00378149cc4-kube-api-access-jrvwl\" (UniqueName: \"kubernetes.io/projected/16761f60-662e-4eb1-a247-b00378149cc4-kube-api-access-jrvwl\") pod \"16761f60-662e-4eb1-a247-b00378149cc4\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " Oct 31 19:56:57.317627 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.317604 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16761f60-662e-4eb1-a247-b00378149cc4-istio-token\" (UniqueName: \"kubernetes.io/projected/16761f60-662e-4eb1-a247-b00378149cc4-istio-token\") pod \"16761f60-662e-4eb1-a247-b00378149cc4\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " Oct 31 19:56:57.317978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.317648 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-credential-socket\") pod \"16761f60-662e-4eb1-a247-b00378149cc4\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " Oct 31 19:56:57.317978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.317736 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-envoy\") pod \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " Oct 31 19:56:57.317978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.317783 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-workload-socket\") pod \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " Oct 31 19:56:57.317978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.317844 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-workload-certs\") pod \"16761f60-662e-4eb1-a247-b00378149cc4\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " Oct 31 19:56:57.317978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.317881 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-credential-socket\") pod \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " Oct 31 19:56:57.317978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.317954 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-workload-certs\") pod \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\" (UID: \"cc8a6def-fe67-42b8-aea6-b3cd2399def1\") " Oct 31 19:56:57.318278 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.318080 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-istio-envoy\") pod \"16761f60-662e-4eb1-a247-b00378149cc4\" (UID: \"16761f60-662e-4eb1-a247-b00378149cc4\") " Oct 31 19:56:57.318829 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.318807 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.318982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.318833 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16761f60-662e-4eb1-a247-b00378149cc4-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.325497 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.325235 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-workload-socket" pod "cc8a6def-fe67-42b8-aea6-b3cd2399def1" (UID: "cc8a6def-fe67-42b8-aea6-b3cd2399def1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:57.325497 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.325389 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-credential-socket" pod "cc8a6def-fe67-42b8-aea6-b3cd2399def1" (UID: "cc8a6def-fe67-42b8-aea6-b3cd2399def1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:57.325699 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.325588 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-credential-socket" pod "16761f60-662e-4eb1-a247-b00378149cc4" (UID: "16761f60-662e-4eb1-a247-b00378149cc4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:57.326818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.325850 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-workload-certs" pod "16761f60-662e-4eb1-a247-b00378149cc4" (UID: "16761f60-662e-4eb1-a247-b00378149cc4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:57.326818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.326077 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-workload-certs" pod "cc8a6def-fe67-42b8-aea6-b3cd2399def1" (UID: "cc8a6def-fe67-42b8-aea6-b3cd2399def1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:57.326818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.326403 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-istio-data" pod "16761f60-662e-4eb1-a247-b00378149cc4" (UID: "16761f60-662e-4eb1-a247-b00378149cc4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:57.326818 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.326546 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istiod-ca-cert" pod "cc8a6def-fe67-42b8-aea6-b3cd2399def1" (UID: "cc8a6def-fe67-42b8-aea6-b3cd2399def1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:56:57.327436 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.327415 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-data" pod "cc8a6def-fe67-42b8-aea6-b3cd2399def1" (UID: "cc8a6def-fe67-42b8-aea6-b3cd2399def1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:57.327752 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.327725 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-envoy" pod "cc8a6def-fe67-42b8-aea6-b3cd2399def1" (UID: "cc8a6def-fe67-42b8-aea6-b3cd2399def1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:57.330689 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.328786 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-token" pod "cc8a6def-fe67-42b8-aea6-b3cd2399def1" (UID: "cc8a6def-fe67-42b8-aea6-b3cd2399def1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:56:57.330689 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.329204 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8a6def-fe67-42b8-aea6-b3cd2399def1-kube-api-access-dz6ts" pod "cc8a6def-fe67-42b8-aea6-b3cd2399def1" (UID: "cc8a6def-fe67-42b8-aea6-b3cd2399def1"). InnerVolumeSpecName "kube-api-access-dz6ts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:56:57.332051 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.331056 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc"] Oct 31 19:56:57.335154 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.335137 4910 scope.go:122] "RemoveContainer" containerID="3333df1718d6f7235bfac8450c3c102e1e1de86bd13998a7f2a32a7cc5213b4d" Oct 31 19:56:57.336659 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.336638 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16761f60-662e-4eb1-a247-b00378149cc4-istio-podinfo" pod "16761f60-662e-4eb1-a247-b00378149cc4" (UID: "16761f60-662e-4eb1-a247-b00378149cc4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:56:57.337334 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.337308 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-istio-envoy" pod "16761f60-662e-4eb1-a247-b00378149cc4" (UID: "16761f60-662e-4eb1-a247-b00378149cc4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:56:57.337334 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.337324 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16761f60-662e-4eb1-a247-b00378149cc4-istio-token" pod "16761f60-662e-4eb1-a247-b00378149cc4" (UID: "16761f60-662e-4eb1-a247-b00378149cc4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:56:57.337640 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.337619 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-n7xkc"] Oct 31 19:56:57.338351 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.338325 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16761f60-662e-4eb1-a247-b00378149cc4-kube-api-access-jrvwl" pod "16761f60-662e-4eb1-a247-b00378149cc4" (UID: "16761f60-662e-4eb1-a247-b00378149cc4"). InnerVolumeSpecName "kube-api-access-jrvwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:56:57.339310 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.339289 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-podinfo" pod "cc8a6def-fe67-42b8-aea6-b3cd2399def1" (UID: "cc8a6def-fe67-42b8-aea6-b3cd2399def1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:56:57.352536 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.352513 4910 scope.go:122] "RemoveContainer" containerID="522881406a390a05402383a2d7dba0ae41adfe62527712ca787e5c2ecc06a5c5" Oct 31 19:56:57.353075 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:56:57.353050 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"522881406a390a05402383a2d7dba0ae41adfe62527712ca787e5c2ecc06a5c5\": container with ID starting with 522881406a390a05402383a2d7dba0ae41adfe62527712ca787e5c2ecc06a5c5 not found: ID does not exist" containerID="522881406a390a05402383a2d7dba0ae41adfe62527712ca787e5c2ecc06a5c5" Oct 31 19:56:57.353146 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.353084 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"522881406a390a05402383a2d7dba0ae41adfe62527712ca787e5c2ecc06a5c5"} err="failed to get container status \"522881406a390a05402383a2d7dba0ae41adfe62527712ca787e5c2ecc06a5c5\": rpc error: code = NotFound desc = could not find container \"522881406a390a05402383a2d7dba0ae41adfe62527712ca787e5c2ecc06a5c5\": container with ID starting with 522881406a390a05402383a2d7dba0ae41adfe62527712ca787e5c2ecc06a5c5 not found: ID does not exist" Oct 31 19:56:57.353146 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.353116 4910 scope.go:122] "RemoveContainer" containerID="3333df1718d6f7235bfac8450c3c102e1e1de86bd13998a7f2a32a7cc5213b4d" Oct 31 19:56:57.353640 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:56:57.353616 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3333df1718d6f7235bfac8450c3c102e1e1de86bd13998a7f2a32a7cc5213b4d\": container with ID starting with 3333df1718d6f7235bfac8450c3c102e1e1de86bd13998a7f2a32a7cc5213b4d not found: ID does not exist" containerID="3333df1718d6f7235bfac8450c3c102e1e1de86bd13998a7f2a32a7cc5213b4d" Oct 31 19:56:57.353744 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.353643 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3333df1718d6f7235bfac8450c3c102e1e1de86bd13998a7f2a32a7cc5213b4d"} err="failed to get container status \"3333df1718d6f7235bfac8450c3c102e1e1de86bd13998a7f2a32a7cc5213b4d\": rpc error: code = NotFound desc = could not find container \"3333df1718d6f7235bfac8450c3c102e1e1de86bd13998a7f2a32a7cc5213b4d\": container with ID starting with 3333df1718d6f7235bfac8450c3c102e1e1de86bd13998a7f2a32a7cc5213b4d not found: ID does not exist" Oct 31 19:56:57.353744 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.353660 4910 scope.go:122] "RemoveContainer" containerID="22b01f8018ba6948f24c94d48d93246192f7e9fcd4f1aeb397b1994908b5d23f" Oct 31 19:56:57.364456 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.364439 4910 scope.go:122] "RemoveContainer" containerID="22b01f8018ba6948f24c94d48d93246192f7e9fcd4f1aeb397b1994908b5d23f" Oct 31 19:56:57.364774 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:56:57.364741 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b01f8018ba6948f24c94d48d93246192f7e9fcd4f1aeb397b1994908b5d23f\": container with ID starting with 22b01f8018ba6948f24c94d48d93246192f7e9fcd4f1aeb397b1994908b5d23f not found: ID does not exist" containerID="22b01f8018ba6948f24c94d48d93246192f7e9fcd4f1aeb397b1994908b5d23f" Oct 31 19:56:57.364864 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.364772 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b01f8018ba6948f24c94d48d93246192f7e9fcd4f1aeb397b1994908b5d23f"} err="failed to get container status \"22b01f8018ba6948f24c94d48d93246192f7e9fcd4f1aeb397b1994908b5d23f\": rpc error: code = NotFound desc = could not find container \"22b01f8018ba6948f24c94d48d93246192f7e9fcd4f1aeb397b1994908b5d23f\": container with ID starting with 22b01f8018ba6948f24c94d48d93246192f7e9fcd4f1aeb397b1994908b5d23f not found: ID does not exist" Oct 31 19:56:57.420686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.420587 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.420686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.420617 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.420686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.420628 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz6ts\" (UniqueName: \"kubernetes.io/projected/cc8a6def-fe67-42b8-aea6-b3cd2399def1-kube-api-access-dz6ts\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.420686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.420638 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.420686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.420647 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.420686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.420655 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16761f60-662e-4eb1-a247-b00378149cc4-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.420686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.420665 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.420686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.420674 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrvwl\" (UniqueName: \"kubernetes.io/projected/16761f60-662e-4eb1-a247-b00378149cc4-kube-api-access-jrvwl\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.420686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.420682 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16761f60-662e-4eb1-a247-b00378149cc4-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.420686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.420691 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.421139 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.420704 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.421139 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.420713 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.421139 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.420726 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.421139 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.420739 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.421139 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.420751 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8a6def-fe67-42b8-aea6-b3cd2399def1-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.421139 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.420782 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16761f60-662e-4eb1-a247-b00378149cc4-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:56:57.612382 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.612354 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48"] Oct 31 19:56:57.619347 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.619317 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-k8p48"] Oct 31 19:56:57.625883 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.625848 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr"] Oct 31 19:56:57.632355 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.632329 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-qvsmr"] Oct 31 19:56:57.757898 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.757810 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16761f60-662e-4eb1-a247-b00378149cc4" path="/var/lib/kubelet/pods/16761f60-662e-4eb1-a247-b00378149cc4/volumes" Oct 31 19:56:57.758484 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.758461 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bff40e4-ffc5-4104-8aed-4ff06167396c" path="/var/lib/kubelet/pods/9bff40e4-ffc5-4104-8aed-4ff06167396c/volumes" Oct 31 19:56:57.759053 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:56:57.759032 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc8a6def-fe67-42b8-aea6-b3cd2399def1" path="/var/lib/kubelet/pods/cc8a6def-fe67-42b8-aea6-b3cd2399def1/volumes" Oct 31 19:57:06.268106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:06.268069 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg" Oct 31 19:57:06.268559 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:06.268161 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp" Oct 31 19:57:09.393839 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.393792 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4"] Oct 31 19:57:09.394879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.394841 4910 cpu_manager.go:414] "containerMap: removing container" podUID="16761f60-662e-4eb1-a247-b00378149cc4" containerName="istio-proxy" Oct 31 19:57:09.394879 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.394871 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16761f60-662e-4eb1-a247-b00378149cc4" containerName="istio-proxy" Oct 31 19:57:09.395091 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.394890 4910 cpu_manager.go:414] "containerMap: removing container" podUID="16761f60-662e-4eb1-a247-b00378149cc4" containerName="istio-proxy" Oct 31 19:57:09.395091 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.394898 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16761f60-662e-4eb1-a247-b00378149cc4" containerName="istio-proxy" Oct 31 19:57:09.395091 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.394967 4910 cpu_manager.go:414] "containerMap: removing container" podUID="cc8a6def-fe67-42b8-aea6-b3cd2399def1" containerName="istio-proxy" Oct 31 19:57:09.395091 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.394975 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc8a6def-fe67-42b8-aea6-b3cd2399def1" containerName="istio-proxy" Oct 31 19:57:09.395091 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.394996 4910 cpu_manager.go:414] "containerMap: removing container" podUID="9bff40e4-ffc5-4104-8aed-4ff06167396c" containerName="istio-proxy" Oct 31 19:57:09.395091 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.395003 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bff40e4-ffc5-4104-8aed-4ff06167396c" containerName="istio-proxy" Oct 31 19:57:09.395477 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.395211 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc8a6def-fe67-42b8-aea6-b3cd2399def1" containerName="istio-proxy" Oct 31 19:57:09.395477 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.395226 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bff40e4-ffc5-4104-8aed-4ff06167396c" containerName="istio-proxy" Oct 31 19:57:09.395477 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.395246 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="16761f60-662e-4eb1-a247-b00378149cc4" containerName="istio-proxy" Oct 31 19:57:09.395477 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.395257 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="16761f60-662e-4eb1-a247-b00378149cc4" containerName="istio-proxy" Oct 31 19:57:09.401751 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.401719 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4" Oct 31 19:57:09.406368 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.406340 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4"] Oct 31 19:57:09.406838 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.406816 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-1f2781-serving-cert\"" Oct 31 19:57:09.539932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.536702 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4w4h\" (UniqueName: \"kubernetes.io/projected/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-kube-api-access-x4w4h\") pod \"maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4\" (UID: \"1031afac-c58c-48e9-9ef6-b6fa69f0f29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4" Oct 31 19:57:09.539932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.536770 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4\" (UID: \"1031afac-c58c-48e9-9ef6-b6fa69f0f29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4" Oct 31 19:57:09.539932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.536851 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-metrics-certs\") pod \"maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4\" (UID: \"1031afac-c58c-48e9-9ef6-b6fa69f0f29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4" Oct 31 19:57:09.539932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.536941 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4\" (UID: \"1031afac-c58c-48e9-9ef6-b6fa69f0f29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4" Oct 31 19:57:09.638139 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.638100 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4\" (UID: \"1031afac-c58c-48e9-9ef6-b6fa69f0f29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4" Oct 31 19:57:09.638341 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.638321 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4w4h\" (UniqueName: \"kubernetes.io/projected/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-kube-api-access-x4w4h\") pod \"maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4\" (UID: \"1031afac-c58c-48e9-9ef6-b6fa69f0f29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4" Oct 31 19:57:09.638429 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.638368 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4\" (UID: \"1031afac-c58c-48e9-9ef6-b6fa69f0f29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4" Oct 31 19:57:09.638486 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.638459 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-metrics-certs\") pod \"maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4\" (UID: \"1031afac-c58c-48e9-9ef6-b6fa69f0f29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4" Oct 31 19:57:09.639097 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.639069 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4\" (UID: \"1031afac-c58c-48e9-9ef6-b6fa69f0f29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4" Oct 31 19:57:09.641011 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.640987 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-metrics-certs\") pod \"maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4\" (UID: \"1031afac-c58c-48e9-9ef6-b6fa69f0f29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4" Oct 31 19:57:09.641254 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.641232 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4\" (UID: \"1031afac-c58c-48e9-9ef6-b6fa69f0f29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4" Oct 31 19:57:09.651995 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.651927 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4w4h\" (UniqueName: \"kubernetes.io/projected/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-kube-api-access-x4w4h\") pod \"maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4\" (UID: \"1031afac-c58c-48e9-9ef6-b6fa69f0f29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4" Oct 31 19:57:09.729082 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:09.729046 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4" Oct 31 19:57:10.140029 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:10.139958 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c25c0405-s9h8f"] Oct 31 19:57:10.216821 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:10.216738 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c25c0405-s9h8f"] Oct 31 19:57:10.228663 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:10.228595 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4"] Oct 31 19:57:10.243739 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:57:10.243698 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1031afac_c58c_48e9_9ef6_b6fa69f0f29e.slice/crio-783baeb587eeeeceaa2fcaa4c655d88f73e8713714e3ad63539ad906d2b4f2e3 WatchSource:0}: Error finding container 783baeb587eeeeceaa2fcaa4c655d88f73e8713714e3ad63539ad906d2b4f2e3: Status 404 returned error can't find the container with id 783baeb587eeeeceaa2fcaa4c655d88f73e8713714e3ad63539ad906d2b4f2e3 Oct 31 19:57:10.482858 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:10.482765 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4" event={"ID":"1031afac-c58c-48e9-9ef6-b6fa69f0f29e","Type":"ContainerStarted","Data":"783baeb587eeeeceaa2fcaa4c655d88f73e8713714e3ad63539ad906d2b4f2e3"} Oct 31 19:57:11.495403 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:11.495367 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4" event={"ID":"1031afac-c58c-48e9-9ef6-b6fa69f0f29e","Type":"ContainerStarted","Data":"3ee8fee18463d8bd6a0bf36fd32a40bc00b6d01c806cfe472f296b98d66bd14a"} Oct 31 19:57:11.495836 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:11.495493 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4" Oct 31 19:57:11.518520 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:11.518471 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4" podStartSLOduration=2.099153532 podStartE2EDuration="2.518455848s" totalImagesPullingTime="419.302316ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:57:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:57:11.516437878 +0000 UTC m=+2265.879258251" watchObservedRunningTime="2026-10-31 19:57:11.518455848 +0000 UTC m=+2265.881276222" Oct 31 19:57:11.758956 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:11.758812 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bc602c5-0eb7-4b12-8821-961094ba097d" path="/var/lib/kubelet/pods/5bc602c5-0eb7-4b12-8821-961094ba097d/volumes" Oct 31 19:57:12.714830 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:57:12.714789 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:57:17.508279 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:17.508244 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4" Oct 31 19:57:28.211981 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:28.211907 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg"] Oct 31 19:57:28.226078 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:28.226043 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp"] Oct 31 19:57:31.096277 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:31.096240 4910 scope.go:122] "RemoveContainer" containerID="f94c942614dc91bc71299e7af736e9afc6a921bd26ca15122ccdc10a26c1a5f1" Oct 31 19:57:31.107835 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:31.107815 4910 scope.go:122] "RemoveContainer" containerID="f5eb1675d3d5a826d29ccdccf56cf13206179c9e005867d6f166c5bf55ab7362" Oct 31 19:57:31.118236 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:31.118215 4910 scope.go:122] "RemoveContainer" containerID="02c8e866d7404627aea4c9ebe53ac88fb863f7478a988313b31894abf8afa24b" Oct 31 19:57:31.130015 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:31.129983 4910 scope.go:122] "RemoveContainer" containerID="3a9ec0241f40c48629c94ee14d8fbe3fcfda70d4efb6e9ca798fa4c7acb3b34e" Oct 31 19:57:31.141939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:31.141900 4910 scope.go:122] "RemoveContainer" containerID="b9ccb137c1e6af7d2ebce38a00b1f3094b32cb2b861b0994b665aebfffc2c004" Oct 31 19:57:31.153312 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:31.153284 4910 scope.go:122] "RemoveContainer" containerID="a510a1ed1b0c4fffe466c3ad147988f645d85aa797ec1402348dbb1174bf9ad3" Oct 31 19:57:31.163154 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:31.163127 4910 scope.go:122] "RemoveContainer" containerID="4d844ca9272105dd46a24bf49e6250d25cfa4eee4cca9b5c49758fb8e5391215" Oct 31 19:57:31.173079 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:31.173056 4910 scope.go:122] "RemoveContainer" containerID="a16125cd51034e99dc1721001f74d87ca55f46e79dd1cbab3452162c339e9297" Oct 31 19:57:33.212908 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.212847 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg" podUID="d3572ce0-4758-4745-8312-3b9bcc942762" containerName="payload-processing" containerID="cri-o://334b1ef01583f54760836d44f07664013e7382f27b24db697fde8ec61a506005" gracePeriod=25 Oct 31 19:57:33.227348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.227305 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp" podUID="2f0e2560-f3d7-4ccc-b139-3f8423f667a1" containerName="payload-pre-processing" containerID="cri-o://78a8d565f89faa69d7db046237af16d2a053c4907da51db36e9c4b6654d55db7" gracePeriod=25 Oct 31 19:57:33.553455 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.553422 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg" Oct 31 19:57:33.606426 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.606400 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp" Oct 31 19:57:33.646268 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.646233 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3572ce0-4758-4745-8312-3b9bcc942762-kube-api-access-4dznn\" (UniqueName: \"kubernetes.io/projected/d3572ce0-4758-4745-8312-3b9bcc942762-kube-api-access-4dznn\") pod \"d3572ce0-4758-4745-8312-3b9bcc942762\" (UID: \"d3572ce0-4758-4745-8312-3b9bcc942762\") " Oct 31 19:57:33.646456 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.646342 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f0e2560-f3d7-4ccc-b139-3f8423f667a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0e2560-f3d7-4ccc-b139-3f8423f667a1-plugins-config-volume\") pod \"2f0e2560-f3d7-4ccc-b139-3f8423f667a1\" (UID: \"2f0e2560-f3d7-4ccc-b139-3f8423f667a1\") " Oct 31 19:57:33.646456 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.646391 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3572ce0-4758-4745-8312-3b9bcc942762-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3572ce0-4758-4745-8312-3b9bcc942762-plugins-config-volume\") pod \"d3572ce0-4758-4745-8312-3b9bcc942762\" (UID: \"d3572ce0-4758-4745-8312-3b9bcc942762\") " Oct 31 19:57:33.646690 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.646674 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f0e2560-f3d7-4ccc-b139-3f8423f667a1-kube-api-access-mflpb\" (UniqueName: \"kubernetes.io/projected/2f0e2560-f3d7-4ccc-b139-3f8423f667a1-kube-api-access-mflpb\") pod \"2f0e2560-f3d7-4ccc-b139-3f8423f667a1\" (UID: \"2f0e2560-f3d7-4ccc-b139-3f8423f667a1\") " Oct 31 19:57:33.647396 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.647369 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f0e2560-f3d7-4ccc-b139-3f8423f667a1-plugins-config-volume" pod "2f0e2560-f3d7-4ccc-b139-3f8423f667a1" (UID: "2f0e2560-f3d7-4ccc-b139-3f8423f667a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:57:33.647994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.647967 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3572ce0-4758-4745-8312-3b9bcc942762-plugins-config-volume" pod "d3572ce0-4758-4745-8312-3b9bcc942762" (UID: "d3572ce0-4758-4745-8312-3b9bcc942762"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:57:33.650095 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.650062 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f0e2560-f3d7-4ccc-b139-3f8423f667a1-kube-api-access-mflpb" pod "2f0e2560-f3d7-4ccc-b139-3f8423f667a1" (UID: "2f0e2560-f3d7-4ccc-b139-3f8423f667a1"). InnerVolumeSpecName "kube-api-access-mflpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:57:33.650333 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.650313 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3572ce0-4758-4745-8312-3b9bcc942762-kube-api-access-4dznn" pod "d3572ce0-4758-4745-8312-3b9bcc942762" (UID: "d3572ce0-4758-4745-8312-3b9bcc942762"). InnerVolumeSpecName "kube-api-access-4dznn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:57:33.749096 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.749016 4910 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f0e2560-f3d7-4ccc-b139-3f8423f667a1-plugins-config-volume\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:57:33.749096 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.749046 4910 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3572ce0-4758-4745-8312-3b9bcc942762-plugins-config-volume\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:57:33.749096 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.749056 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mflpb\" (UniqueName: \"kubernetes.io/projected/2f0e2560-f3d7-4ccc-b139-3f8423f667a1-kube-api-access-mflpb\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:57:33.749096 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.749078 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dznn\" (UniqueName: \"kubernetes.io/projected/d3572ce0-4758-4745-8312-3b9bcc942762-kube-api-access-4dznn\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:57:33.817955 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.817889 4910 generic.go:356] "Generic (PLEG): container finished" podID="2f0e2560-f3d7-4ccc-b139-3f8423f667a1" containerID="78a8d565f89faa69d7db046237af16d2a053c4907da51db36e9c4b6654d55db7" exitCode=0 Oct 31 19:57:33.817955 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.817942 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp" event={"ID":"2f0e2560-f3d7-4ccc-b139-3f8423f667a1","Type":"ContainerDied","Data":"78a8d565f89faa69d7db046237af16d2a053c4907da51db36e9c4b6654d55db7"} Oct 31 19:57:33.818182 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.817972 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp" Oct 31 19:57:33.818182 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.818039 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp" event={"ID":"2f0e2560-f3d7-4ccc-b139-3f8423f667a1","Type":"ContainerDied","Data":"550fdc8c861ac91ce34e84e4db01a7e2232256c02cd8e30efd6867200dc7fbba"} Oct 31 19:57:33.818182 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.818070 4910 scope.go:122] "RemoveContainer" containerID="78a8d565f89faa69d7db046237af16d2a053c4907da51db36e9c4b6654d55db7" Oct 31 19:57:33.820627 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.820520 4910 generic.go:356] "Generic (PLEG): container finished" podID="d3572ce0-4758-4745-8312-3b9bcc942762" containerID="334b1ef01583f54760836d44f07664013e7382f27b24db697fde8ec61a506005" exitCode=0 Oct 31 19:57:33.820763 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.820630 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg" event={"ID":"d3572ce0-4758-4745-8312-3b9bcc942762","Type":"ContainerDied","Data":"334b1ef01583f54760836d44f07664013e7382f27b24db697fde8ec61a506005"} Oct 31 19:57:33.820763 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.820681 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg" event={"ID":"d3572ce0-4758-4745-8312-3b9bcc942762","Type":"ContainerDied","Data":"81504bac03d08fbd5cce4a45ac72da78dcf45775e3d0fdc397f34ea2d5294da0"} Oct 31 19:57:33.820878 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.820772 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg" Oct 31 19:57:33.831236 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.831215 4910 scope.go:122] "RemoveContainer" containerID="78a8d565f89faa69d7db046237af16d2a053c4907da51db36e9c4b6654d55db7" Oct 31 19:57:33.831557 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:57:33.831535 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78a8d565f89faa69d7db046237af16d2a053c4907da51db36e9c4b6654d55db7\": container with ID starting with 78a8d565f89faa69d7db046237af16d2a053c4907da51db36e9c4b6654d55db7 not found: ID does not exist" containerID="78a8d565f89faa69d7db046237af16d2a053c4907da51db36e9c4b6654d55db7" Oct 31 19:57:33.831638 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.831562 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78a8d565f89faa69d7db046237af16d2a053c4907da51db36e9c4b6654d55db7"} err="failed to get container status \"78a8d565f89faa69d7db046237af16d2a053c4907da51db36e9c4b6654d55db7\": rpc error: code = NotFound desc = could not find container \"78a8d565f89faa69d7db046237af16d2a053c4907da51db36e9c4b6654d55db7\": container with ID starting with 78a8d565f89faa69d7db046237af16d2a053c4907da51db36e9c4b6654d55db7 not found: ID does not exist" Oct 31 19:57:33.831638 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.831579 4910 scope.go:122] "RemoveContainer" containerID="334b1ef01583f54760836d44f07664013e7382f27b24db697fde8ec61a506005" Oct 31 19:57:33.841205 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.841181 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg"] Oct 31 19:57:33.845035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.845011 4910 scope.go:122] "RemoveContainer" containerID="334b1ef01583f54760836d44f07664013e7382f27b24db697fde8ec61a506005" Oct 31 19:57:33.845370 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:57:33.845345 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"334b1ef01583f54760836d44f07664013e7382f27b24db697fde8ec61a506005\": container with ID starting with 334b1ef01583f54760836d44f07664013e7382f27b24db697fde8ec61a506005 not found: ID does not exist" containerID="334b1ef01583f54760836d44f07664013e7382f27b24db697fde8ec61a506005" Oct 31 19:57:33.845461 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.845380 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"334b1ef01583f54760836d44f07664013e7382f27b24db697fde8ec61a506005"} err="failed to get container status \"334b1ef01583f54760836d44f07664013e7382f27b24db697fde8ec61a506005\": rpc error: code = NotFound desc = could not find container \"334b1ef01583f54760836d44f07664013e7382f27b24db697fde8ec61a506005\": container with ID starting with 334b1ef01583f54760836d44f07664013e7382f27b24db697fde8ec61a506005 not found: ID does not exist" Oct 31 19:57:33.850094 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.850069 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1f2781-76cd6d497f-sbqjg"] Oct 31 19:57:33.859827 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.859805 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp"] Oct 31 19:57:33.866567 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:33.866546 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1f2781-57c8788479-894jp"] Oct 31 19:57:35.758244 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:35.758204 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f0e2560-f3d7-4ccc-b139-3f8423f667a1" path="/var/lib/kubelet/pods/2f0e2560-f3d7-4ccc-b139-3f8423f667a1/volumes" Oct 31 19:57:35.758955 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:35.758895 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3572ce0-4758-4745-8312-3b9bcc942762" path="/var/lib/kubelet/pods/d3572ce0-4758-4745-8312-3b9bcc942762/volumes" Oct 31 19:57:42.715659 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:57:42.715614 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:57:46.163703 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:46.163666 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7008-354e46"] Oct 31 19:57:46.164643 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:46.164608 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d3572ce0-4758-4745-8312-3b9bcc942762" containerName="payload-processing" Oct 31 19:57:46.164643 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:46.164641 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3572ce0-4758-4745-8312-3b9bcc942762" containerName="payload-processing" Oct 31 19:57:46.164816 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:46.164726 4910 cpu_manager.go:414] "containerMap: removing container" podUID="2f0e2560-f3d7-4ccc-b139-3f8423f667a1" containerName="payload-pre-processing" Oct 31 19:57:46.164816 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:46.164734 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f0e2560-f3d7-4ccc-b139-3f8423f667a1" containerName="payload-pre-processing" Oct 31 19:57:46.164885 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:46.164857 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f0e2560-f3d7-4ccc-b139-3f8423f667a1" containerName="payload-pre-processing" Oct 31 19:57:46.164885 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:46.164867 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3572ce0-4758-4745-8312-3b9bcc942762" containerName="payload-processing" Oct 31 19:57:46.170328 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:46.170286 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7008-354e46" Oct 31 19:57:46.174032 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:46.174002 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7008-354e46"] Oct 31 19:57:46.174658 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:46.174633 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4pmld\"" Oct 31 19:57:46.359537 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:46.359496 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw2f4\" (UniqueName: \"kubernetes.io/projected/e312f0b3-d8cc-4128-8038-bf47f3be9f52-kube-api-access-gw2f4\") pod \"test-curl-7008-354e46\" (UID: \"e312f0b3-d8cc-4128-8038-bf47f3be9f52\") " pod="opendatahub/test-curl-7008-354e46" Oct 31 19:57:46.461904 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:46.461811 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw2f4\" (UniqueName: \"kubernetes.io/projected/e312f0b3-d8cc-4128-8038-bf47f3be9f52-kube-api-access-gw2f4\") pod \"test-curl-7008-354e46\" (UID: \"e312f0b3-d8cc-4128-8038-bf47f3be9f52\") " pod="opendatahub/test-curl-7008-354e46" Oct 31 19:57:46.475031 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:46.474990 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw2f4\" (UniqueName: \"kubernetes.io/projected/e312f0b3-d8cc-4128-8038-bf47f3be9f52-kube-api-access-gw2f4\") pod \"test-curl-7008-354e46\" (UID: \"e312f0b3-d8cc-4128-8038-bf47f3be9f52\") " pod="opendatahub/test-curl-7008-354e46" Oct 31 19:57:46.563739 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:46.563702 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7008-354e46" Oct 31 19:57:46.846868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:46.846838 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7008-354e46"] Oct 31 19:57:46.847897 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:57:46.847870 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode312f0b3_d8cc_4128_8038_bf47f3be9f52.slice/crio-54d3e8e2a187bc2cee48b9fa00722a685b5fff7667c0a6ecadbb33cdcba4a03a WatchSource:0}: Error finding container 54d3e8e2a187bc2cee48b9fa00722a685b5fff7667c0a6ecadbb33cdcba4a03a: Status 404 returned error can't find the container with id 54d3e8e2a187bc2cee48b9fa00722a685b5fff7667c0a6ecadbb33cdcba4a03a Oct 31 19:57:47.072765 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:47.072728 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7008-354e46" event={"ID":"e312f0b3-d8cc-4128-8038-bf47f3be9f52","Type":"ContainerStarted","Data":"d1417f210a64cf3346b68c1cd7c3c93c44b26c04cbe3da43a35021d4fd544cbb"} Oct 31 19:57:47.072765 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:47.072774 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7008-354e46" event={"ID":"e312f0b3-d8cc-4128-8038-bf47f3be9f52","Type":"ContainerStarted","Data":"54d3e8e2a187bc2cee48b9fa00722a685b5fff7667c0a6ecadbb33cdcba4a03a"} Oct 31 19:57:47.095788 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:47.095688 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7008-354e46" podStartSLOduration=1.095674662 podStartE2EDuration="1.095674662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:57:47.0925499 +0000 UTC m=+2301.455370272" watchObservedRunningTime="2026-10-31 19:57:47.095674662 +0000 UTC m=+2301.458495034" Oct 31 19:57:47.826505 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:47.826452 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7008-354e46"] Oct 31 19:57:47.831557 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:47.831525 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7008-354e46"] Oct 31 19:57:48.035882 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:48.035839 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-042b89d2-fjctk"] Oct 31 19:57:48.036520 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:48.036501 4910 cpu_manager.go:414] "containerMap: removing container" podUID="e312f0b3-d8cc-4128-8038-bf47f3be9f52" containerName="test-curl-7008-354e46" Oct 31 19:57:48.036520 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:48.036519 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e312f0b3-d8cc-4128-8038-bf47f3be9f52" containerName="test-curl-7008-354e46" Oct 31 19:57:48.036674 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:48.036634 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="e312f0b3-d8cc-4128-8038-bf47f3be9f52" containerName="test-curl-7008-354e46" Oct 31 19:57:48.040323 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:48.040302 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-042b89d2-fjctk" Oct 31 19:57:48.046362 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:48.046311 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:57:48.046505 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:48.046442 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:57:48.050249 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:48.050229 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-042b89d2-fjctk"] Oct 31 19:57:48.187566 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:48.187471 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e565eb1-e5ec-4638-9412-26692a7cac42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-042b89d2-fjctk\" (UID: \"0e565eb1-e5ec-4638-9412-26692a7cac42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-042b89d2-fjctk" Oct 31 19:57:48.187746 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:48.187647 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e565eb1-e5ec-4638-9412-26692a7cac42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-042b89d2-fjctk\" (UID: \"0e565eb1-e5ec-4638-9412-26692a7cac42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-042b89d2-fjctk" Oct 31 19:57:48.187947 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:48.187924 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwrkp\" (UniqueName: \"kubernetes.io/projected/0e565eb1-e5ec-4638-9412-26692a7cac42-kube-api-access-xwrkp\") pod \"maas-api-revoke-sub-042b89d2-fjctk\" (UID: \"0e565eb1-e5ec-4638-9412-26692a7cac42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-042b89d2-fjctk" Oct 31 19:57:48.289693 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:48.289655 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e565eb1-e5ec-4638-9412-26692a7cac42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-042b89d2-fjctk\" (UID: \"0e565eb1-e5ec-4638-9412-26692a7cac42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-042b89d2-fjctk" Oct 31 19:57:48.289893 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:48.289801 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e565eb1-e5ec-4638-9412-26692a7cac42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-042b89d2-fjctk\" (UID: \"0e565eb1-e5ec-4638-9412-26692a7cac42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-042b89d2-fjctk" Oct 31 19:57:48.290109 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:48.290089 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwrkp\" (UniqueName: \"kubernetes.io/projected/0e565eb1-e5ec-4638-9412-26692a7cac42-kube-api-access-xwrkp\") pod \"maas-api-revoke-sub-042b89d2-fjctk\" (UID: \"0e565eb1-e5ec-4638-9412-26692a7cac42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-042b89d2-fjctk" Oct 31 19:57:48.290547 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:48.290508 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e565eb1-e5ec-4638-9412-26692a7cac42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-042b89d2-fjctk\" (UID: \"0e565eb1-e5ec-4638-9412-26692a7cac42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-042b89d2-fjctk" Oct 31 19:57:48.292128 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:48.292108 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e565eb1-e5ec-4638-9412-26692a7cac42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-042b89d2-fjctk\" (UID: \"0e565eb1-e5ec-4638-9412-26692a7cac42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-042b89d2-fjctk" Oct 31 19:57:48.303468 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:48.303439 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwrkp\" (UniqueName: \"kubernetes.io/projected/0e565eb1-e5ec-4638-9412-26692a7cac42-kube-api-access-xwrkp\") pod \"maas-api-revoke-sub-042b89d2-fjctk\" (UID: \"0e565eb1-e5ec-4638-9412-26692a7cac42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-042b89d2-fjctk" Oct 31 19:57:48.361121 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:48.361088 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-042b89d2-fjctk" Oct 31 19:57:48.644229 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:57:48.644190 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e565eb1_e5ec_4638_9412_26692a7cac42.slice/crio-bdb432df228b1ff0cc9f8e10cce7ec35221a07478d7650da2e597a228e4b6aac WatchSource:0}: Error finding container bdb432df228b1ff0cc9f8e10cce7ec35221a07478d7650da2e597a228e4b6aac: Status 404 returned error can't find the container with id bdb432df228b1ff0cc9f8e10cce7ec35221a07478d7650da2e597a228e4b6aac Oct 31 19:57:48.645323 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:48.645299 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-042b89d2-fjctk"] Oct 31 19:57:49.093001 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:49.092956 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-042b89d2-fjctk" event={"ID":"0e565eb1-e5ec-4638-9412-26692a7cac42","Type":"ContainerStarted","Data":"66e9137467a4dee18ef419653f286177eac76183280dde096e433ffac03cd602"} Oct 31 19:57:49.093487 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:49.093030 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-042b89d2-fjctk" event={"ID":"0e565eb1-e5ec-4638-9412-26692a7cac42","Type":"ContainerStarted","Data":"bdb432df228b1ff0cc9f8e10cce7ec35221a07478d7650da2e597a228e4b6aac"} Oct 31 19:57:49.093487 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:49.093031 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7008-354e46" podUID="e312f0b3-d8cc-4128-8038-bf47f3be9f52" containerName="test-curl-7008-354e46" containerID="cri-o://d1417f210a64cf3346b68c1cd7c3c93c44b26c04cbe3da43a35021d4fd544cbb" gracePeriod=2 Oct 31 19:57:49.096062 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:57:49.096034 4910 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7008-354e46\" is forbidden: User \"system:node:ip-10-0-142-116.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-116.us-east-2.compute.internal' and this object" podUID="e312f0b3-d8cc-4128-8038-bf47f3be9f52" pod="opendatahub/test-curl-7008-354e46" Oct 31 19:57:49.116287 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:49.116239 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-042b89d2-fjctk" podStartSLOduration=1.116226435 podStartE2EDuration="1.116226435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:57:49.11431285 +0000 UTC m=+2303.477133223" watchObservedRunningTime="2026-10-31 19:57:49.116226435 +0000 UTC m=+2303.479046799" Oct 31 19:57:49.117348 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:57:49.117324 4910 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7008-354e46\" is forbidden: User \"system:node:ip-10-0-142-116.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-116.us-east-2.compute.internal' and this object" podUID="e312f0b3-d8cc-4128-8038-bf47f3be9f52" pod="opendatahub/test-curl-7008-354e46" Oct 31 19:57:50.102081 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:50.102040 4910 generic.go:356] "Generic (PLEG): container finished" podID="0e565eb1-e5ec-4638-9412-26692a7cac42" containerID="66e9137467a4dee18ef419653f286177eac76183280dde096e433ffac03cd602" exitCode=0 Oct 31 19:57:50.102479 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:50.102118 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-042b89d2-fjctk" event={"ID":"0e565eb1-e5ec-4638-9412-26692a7cac42","Type":"ContainerDied","Data":"66e9137467a4dee18ef419653f286177eac76183280dde096e433ffac03cd602"} Oct 31 19:57:51.363536 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:51.363467 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-042b89d2-fjctk" Oct 31 19:57:51.407671 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:51.407642 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7008-354e46" Oct 31 19:57:51.457655 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:51.457622 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e565eb1-e5ec-4638-9412-26692a7cac42-kube-api-access-xwrkp\" (UniqueName: \"kubernetes.io/projected/0e565eb1-e5ec-4638-9412-26692a7cac42-kube-api-access-xwrkp\") pod \"0e565eb1-e5ec-4638-9412-26692a7cac42\" (UID: \"0e565eb1-e5ec-4638-9412-26692a7cac42\") " Oct 31 19:57:51.457833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:51.457693 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e565eb1-e5ec-4638-9412-26692a7cac42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e565eb1-e5ec-4638-9412-26692a7cac42-maas-api-openshift-ca\") pod \"0e565eb1-e5ec-4638-9412-26692a7cac42\" (UID: \"0e565eb1-e5ec-4638-9412-26692a7cac42\") " Oct 31 19:57:51.457833 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:51.457764 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e565eb1-e5ec-4638-9412-26692a7cac42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e565eb1-e5ec-4638-9412-26692a7cac42-maas-api-xks-ca\") pod \"0e565eb1-e5ec-4638-9412-26692a7cac42\" (UID: \"0e565eb1-e5ec-4638-9412-26692a7cac42\") " Oct 31 19:57:51.457942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:51.457882 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e312f0b3-d8cc-4128-8038-bf47f3be9f52-kube-api-access-gw2f4\" (UniqueName: \"kubernetes.io/projected/e312f0b3-d8cc-4128-8038-bf47f3be9f52-kube-api-access-gw2f4\") pod \"e312f0b3-d8cc-4128-8038-bf47f3be9f52\" (UID: \"e312f0b3-d8cc-4128-8038-bf47f3be9f52\") " Oct 31 19:57:51.458181 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:51.458151 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e565eb1-e5ec-4638-9412-26692a7cac42-maas-api-openshift-ca" pod "0e565eb1-e5ec-4638-9412-26692a7cac42" (UID: "0e565eb1-e5ec-4638-9412-26692a7cac42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:57:51.458819 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:51.458792 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e565eb1-e5ec-4638-9412-26692a7cac42-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:57:51.460034 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:51.460006 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e565eb1-e5ec-4638-9412-26692a7cac42-maas-api-xks-ca" pod "0e565eb1-e5ec-4638-9412-26692a7cac42" (UID: "0e565eb1-e5ec-4638-9412-26692a7cac42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:57:51.460141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:51.460057 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e565eb1-e5ec-4638-9412-26692a7cac42-kube-api-access-xwrkp" pod "0e565eb1-e5ec-4638-9412-26692a7cac42" (UID: "0e565eb1-e5ec-4638-9412-26692a7cac42"). InnerVolumeSpecName "kube-api-access-xwrkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:57:51.460141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:51.460076 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e312f0b3-d8cc-4128-8038-bf47f3be9f52-kube-api-access-gw2f4" pod "e312f0b3-d8cc-4128-8038-bf47f3be9f52" (UID: "e312f0b3-d8cc-4128-8038-bf47f3be9f52"). InnerVolumeSpecName "kube-api-access-gw2f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:57:51.560176 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:51.560140 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwrkp\" (UniqueName: \"kubernetes.io/projected/0e565eb1-e5ec-4638-9412-26692a7cac42-kube-api-access-xwrkp\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:57:51.560176 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:51.560171 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e565eb1-e5ec-4638-9412-26692a7cac42-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:57:51.560176 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:51.560181 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw2f4\" (UniqueName: \"kubernetes.io/projected/e312f0b3-d8cc-4128-8038-bf47f3be9f52-kube-api-access-gw2f4\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:57:51.756466 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:51.756385 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e312f0b3-d8cc-4128-8038-bf47f3be9f52" path="/var/lib/kubelet/pods/e312f0b3-d8cc-4128-8038-bf47f3be9f52/volumes" Oct 31 19:57:52.168161 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:52.168114 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-042b89d2-fjctk" event={"ID":"0e565eb1-e5ec-4638-9412-26692a7cac42","Type":"ContainerDied","Data":"bdb432df228b1ff0cc9f8e10cce7ec35221a07478d7650da2e597a228e4b6aac"} Oct 31 19:57:52.168378 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:52.168194 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdb432df228b1ff0cc9f8e10cce7ec35221a07478d7650da2e597a228e4b6aac" Oct 31 19:57:52.168378 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:52.168134 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-042b89d2-fjctk" Oct 31 19:57:52.170439 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:52.170403 4910 generic.go:356] "Generic (PLEG): container finished" podID="e312f0b3-d8cc-4128-8038-bf47f3be9f52" containerID="d1417f210a64cf3346b68c1cd7c3c93c44b26c04cbe3da43a35021d4fd544cbb" exitCode=137 Oct 31 19:57:52.170802 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:52.170679 4910 scope.go:122] "RemoveContainer" containerID="d1417f210a64cf3346b68c1cd7c3c93c44b26c04cbe3da43a35021d4fd544cbb" Oct 31 19:57:52.170802 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:52.170759 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7008-354e46" Oct 31 19:57:52.183269 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:52.183249 4910 scope.go:122] "RemoveContainer" containerID="d1417f210a64cf3346b68c1cd7c3c93c44b26c04cbe3da43a35021d4fd544cbb" Oct 31 19:57:52.183603 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:57:52.183584 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1417f210a64cf3346b68c1cd7c3c93c44b26c04cbe3da43a35021d4fd544cbb\": container with ID starting with d1417f210a64cf3346b68c1cd7c3c93c44b26c04cbe3da43a35021d4fd544cbb not found: ID does not exist" containerID="d1417f210a64cf3346b68c1cd7c3c93c44b26c04cbe3da43a35021d4fd544cbb" Oct 31 19:57:52.183667 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:52.183615 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1417f210a64cf3346b68c1cd7c3c93c44b26c04cbe3da43a35021d4fd544cbb"} err="failed to get container status \"d1417f210a64cf3346b68c1cd7c3c93c44b26c04cbe3da43a35021d4fd544cbb\": rpc error: code = NotFound desc = could not find container \"d1417f210a64cf3346b68c1cd7c3c93c44b26c04cbe3da43a35021d4fd544cbb\": container with ID starting with d1417f210a64cf3346b68c1cd7c3c93c44b26c04cbe3da43a35021d4fd544cbb not found: ID does not exist" Oct 31 19:57:58.349993 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:58.349951 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-155ad5a0-8w2wv"] Oct 31 19:57:58.350845 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:58.350820 4910 cpu_manager.go:414] "containerMap: removing container" podUID="0e565eb1-e5ec-4638-9412-26692a7cac42" containerName="revoke-keys" Oct 31 19:57:58.350845 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:58.350846 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e565eb1-e5ec-4638-9412-26692a7cac42" containerName="revoke-keys" Oct 31 19:57:58.351073 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:58.351055 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e565eb1-e5ec-4638-9412-26692a7cac42" containerName="revoke-keys" Oct 31 19:57:58.356188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:58.356168 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-155ad5a0-8w2wv" Oct 31 19:57:58.370044 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:58.370014 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:57:58.370176 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:58.370055 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:57:58.374051 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:58.374019 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-155ad5a0-8w2wv"] Oct 31 19:57:58.398670 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:58.398635 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95eff1aa-9f5f-4009-8dd7-003f1181f586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-155ad5a0-8w2wv\" (UID: \"95eff1aa-9f5f-4009-8dd7-003f1181f586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155ad5a0-8w2wv" Oct 31 19:57:58.398822 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:58.398794 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95eff1aa-9f5f-4009-8dd7-003f1181f586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-155ad5a0-8w2wv\" (UID: \"95eff1aa-9f5f-4009-8dd7-003f1181f586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155ad5a0-8w2wv" Oct 31 19:57:58.399003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:58.398977 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlblb\" (UniqueName: \"kubernetes.io/projected/95eff1aa-9f5f-4009-8dd7-003f1181f586-kube-api-access-dlblb\") pod \"maas-api-revoke-sub-155ad5a0-8w2wv\" (UID: \"95eff1aa-9f5f-4009-8dd7-003f1181f586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155ad5a0-8w2wv" Oct 31 19:57:58.500765 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:58.500727 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlblb\" (UniqueName: \"kubernetes.io/projected/95eff1aa-9f5f-4009-8dd7-003f1181f586-kube-api-access-dlblb\") pod \"maas-api-revoke-sub-155ad5a0-8w2wv\" (UID: \"95eff1aa-9f5f-4009-8dd7-003f1181f586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155ad5a0-8w2wv" Oct 31 19:57:58.500973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:58.500900 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95eff1aa-9f5f-4009-8dd7-003f1181f586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-155ad5a0-8w2wv\" (UID: \"95eff1aa-9f5f-4009-8dd7-003f1181f586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155ad5a0-8w2wv" Oct 31 19:57:58.501018 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:58.500971 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95eff1aa-9f5f-4009-8dd7-003f1181f586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-155ad5a0-8w2wv\" (UID: \"95eff1aa-9f5f-4009-8dd7-003f1181f586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155ad5a0-8w2wv" Oct 31 19:57:58.501539 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:58.501515 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95eff1aa-9f5f-4009-8dd7-003f1181f586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-155ad5a0-8w2wv\" (UID: \"95eff1aa-9f5f-4009-8dd7-003f1181f586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155ad5a0-8w2wv" Oct 31 19:57:58.503265 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:58.503244 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95eff1aa-9f5f-4009-8dd7-003f1181f586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-155ad5a0-8w2wv\" (UID: \"95eff1aa-9f5f-4009-8dd7-003f1181f586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155ad5a0-8w2wv" Oct 31 19:57:58.513054 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:58.513020 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlblb\" (UniqueName: \"kubernetes.io/projected/95eff1aa-9f5f-4009-8dd7-003f1181f586-kube-api-access-dlblb\") pod \"maas-api-revoke-sub-155ad5a0-8w2wv\" (UID: \"95eff1aa-9f5f-4009-8dd7-003f1181f586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-155ad5a0-8w2wv" Oct 31 19:57:58.675207 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:58.675126 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-155ad5a0-8w2wv" Oct 31 19:57:58.979353 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:57:58.979318 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95eff1aa_9f5f_4009_8dd7_003f1181f586.slice/crio-d9bbeff1681f172413320638fd9829f42049d8218216211bccf5ddc3776a34e1 WatchSource:0}: Error finding container d9bbeff1681f172413320638fd9829f42049d8218216211bccf5ddc3776a34e1: Status 404 returned error can't find the container with id d9bbeff1681f172413320638fd9829f42049d8218216211bccf5ddc3776a34e1 Oct 31 19:57:58.979505 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:58.979432 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-155ad5a0-8w2wv"] Oct 31 19:57:59.235255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:57:59.235210 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-155ad5a0-8w2wv" event={"ID":"95eff1aa-9f5f-4009-8dd7-003f1181f586","Type":"ContainerStarted","Data":"d9bbeff1681f172413320638fd9829f42049d8218216211bccf5ddc3776a34e1"} Oct 31 19:58:00.248588 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:00.248545 4910 generic.go:356] "Generic (PLEG): container finished" podID="95eff1aa-9f5f-4009-8dd7-003f1181f586" containerID="383173a8d9b8a3a84a0a66150e41fd92048e0beb4b8f71d675632ae787898861" exitCode=0 Oct 31 19:58:00.249102 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:00.248802 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-155ad5a0-8w2wv" event={"ID":"95eff1aa-9f5f-4009-8dd7-003f1181f586","Type":"ContainerDied","Data":"383173a8d9b8a3a84a0a66150e41fd92048e0beb4b8f71d675632ae787898861"} Oct 31 19:58:01.443238 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:01.443211 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-155ad5a0-8w2wv" Oct 31 19:58:01.578415 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:01.578377 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95eff1aa-9f5f-4009-8dd7-003f1181f586-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95eff1aa-9f5f-4009-8dd7-003f1181f586-maas-api-xks-ca\") pod \"95eff1aa-9f5f-4009-8dd7-003f1181f586\" (UID: \"95eff1aa-9f5f-4009-8dd7-003f1181f586\") " Oct 31 19:58:01.578610 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:01.578540 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95eff1aa-9f5f-4009-8dd7-003f1181f586-kube-api-access-dlblb\" (UniqueName: \"kubernetes.io/projected/95eff1aa-9f5f-4009-8dd7-003f1181f586-kube-api-access-dlblb\") pod \"95eff1aa-9f5f-4009-8dd7-003f1181f586\" (UID: \"95eff1aa-9f5f-4009-8dd7-003f1181f586\") " Oct 31 19:58:01.578610 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:01.578584 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95eff1aa-9f5f-4009-8dd7-003f1181f586-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95eff1aa-9f5f-4009-8dd7-003f1181f586-maas-api-openshift-ca\") pod \"95eff1aa-9f5f-4009-8dd7-003f1181f586\" (UID: \"95eff1aa-9f5f-4009-8dd7-003f1181f586\") " Oct 31 19:58:01.578989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:01.578959 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95eff1aa-9f5f-4009-8dd7-003f1181f586-maas-api-openshift-ca" pod "95eff1aa-9f5f-4009-8dd7-003f1181f586" (UID: "95eff1aa-9f5f-4009-8dd7-003f1181f586"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:58:01.579235 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:01.579216 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95eff1aa-9f5f-4009-8dd7-003f1181f586-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:01.580614 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:01.580590 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95eff1aa-9f5f-4009-8dd7-003f1181f586-maas-api-xks-ca" pod "95eff1aa-9f5f-4009-8dd7-003f1181f586" (UID: "95eff1aa-9f5f-4009-8dd7-003f1181f586"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:58:01.580695 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:01.580662 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95eff1aa-9f5f-4009-8dd7-003f1181f586-kube-api-access-dlblb" pod "95eff1aa-9f5f-4009-8dd7-003f1181f586" (UID: "95eff1aa-9f5f-4009-8dd7-003f1181f586"). InnerVolumeSpecName "kube-api-access-dlblb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:58:01.680861 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:01.680822 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlblb\" (UniqueName: \"kubernetes.io/projected/95eff1aa-9f5f-4009-8dd7-003f1181f586-kube-api-access-dlblb\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:01.680861 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:01.680856 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95eff1aa-9f5f-4009-8dd7-003f1181f586-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:02.267171 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:02.267136 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-155ad5a0-8w2wv" Oct 31 19:58:02.267171 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:02.267161 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-155ad5a0-8w2wv" event={"ID":"95eff1aa-9f5f-4009-8dd7-003f1181f586","Type":"ContainerDied","Data":"d9bbeff1681f172413320638fd9829f42049d8218216211bccf5ddc3776a34e1"} Oct 31 19:58:02.267381 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:02.267211 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9bbeff1681f172413320638fd9829f42049d8218216211bccf5ddc3776a34e1" Oct 31 19:58:11.488483 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:11.488450 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2"] Oct 31 19:58:11.489307 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:11.489281 4910 cpu_manager.go:414] "containerMap: removing container" podUID="95eff1aa-9f5f-4009-8dd7-003f1181f586" containerName="revoke-keys" Oct 31 19:58:11.489307 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:11.489306 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95eff1aa-9f5f-4009-8dd7-003f1181f586" containerName="revoke-keys" Oct 31 19:58:11.489571 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:11.489556 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="95eff1aa-9f5f-4009-8dd7-003f1181f586" containerName="revoke-keys" Oct 31 19:58:11.493599 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:11.493577 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2" Oct 31 19:58:11.501697 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:11.501675 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-01afb5-dockercfg-9jzvm\"" Oct 31 19:58:11.501697 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:11.501676 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-01afb5\"" Oct 31 19:58:11.507274 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:11.507169 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2"] Oct 31 19:58:11.561894 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:11.561853 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js7hj\" (UniqueName: \"kubernetes.io/projected/45fc8f3a-bd26-4da4-abcd-68f875727ff3-kube-api-access-js7hj\") pod \"payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2\" (UID: \"45fc8f3a-bd26-4da4-abcd-68f875727ff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2" Oct 31 19:58:11.562193 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:11.562134 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45fc8f3a-bd26-4da4-abcd-68f875727ff3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2\" (UID: \"45fc8f3a-bd26-4da4-abcd-68f875727ff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2" Oct 31 19:58:11.704938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:11.692334 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45fc8f3a-bd26-4da4-abcd-68f875727ff3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2\" (UID: \"45fc8f3a-bd26-4da4-abcd-68f875727ff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2" Oct 31 19:58:11.705470 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:11.705440 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js7hj\" (UniqueName: \"kubernetes.io/projected/45fc8f3a-bd26-4da4-abcd-68f875727ff3-kube-api-access-js7hj\") pod \"payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2\" (UID: \"45fc8f3a-bd26-4da4-abcd-68f875727ff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2" Oct 31 19:58:11.706322 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:11.699821 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45fc8f3a-bd26-4da4-abcd-68f875727ff3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2\" (UID: \"45fc8f3a-bd26-4da4-abcd-68f875727ff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2" Oct 31 19:58:11.738980 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:11.729845 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js7hj\" (UniqueName: \"kubernetes.io/projected/45fc8f3a-bd26-4da4-abcd-68f875727ff3-kube-api-access-js7hj\") pod \"payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2\" (UID: \"45fc8f3a-bd26-4da4-abcd-68f875727ff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2" Oct 31 19:58:11.826938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:11.824067 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2" Oct 31 19:58:12.240593 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:12.240557 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2"] Oct 31 19:58:12.442325 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:12.442275 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2" event={"ID":"45fc8f3a-bd26-4da4-abcd-68f875727ff3","Type":"ContainerStarted","Data":"ebdfcc9fdbf99a21d9c08a807e242a675368f9d73ef48e1202a3d7412674dd08"} Oct 31 19:58:12.443598 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:12.443573 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2" Oct 31 19:58:12.443810 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:12.443780 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2" event={"ID":"45fc8f3a-bd26-4da4-abcd-68f875727ff3","Type":"ContainerStarted","Data":"455d99c1e98bd8644fafe27c7bf57e9521a2b3d86da728dc340a29dbf4fd8a0d"} Oct 31 19:58:12.466402 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:12.466350 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2" podStartSLOduration=1.466333527 podStartE2EDuration="1.466333527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:58:12.46341638 +0000 UTC m=+2326.826236753" watchObservedRunningTime="2026-10-31 19:58:12.466333527 +0000 UTC m=+2326.829153899" Oct 31 19:58:12.719842 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:58:12.719801 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:58:17.093796 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:17.093757 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bba4c0c7-z89pz"] Oct 31 19:58:17.098057 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:17.098033 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba4c0c7-z89pz" Oct 31 19:58:17.103345 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:17.103309 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:58:17.104774 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:17.104547 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:58:17.112897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:17.112871 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bba4c0c7-z89pz"] Oct 31 19:58:17.142055 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:17.139851 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21dfb59-b7b0-4dfc-a842-00f203c7b9ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bba4c0c7-z89pz\" (UID: \"c21dfb59-b7b0-4dfc-a842-00f203c7b9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba4c0c7-z89pz" Oct 31 19:58:17.142055 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:17.139957 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21dfb59-b7b0-4dfc-a842-00f203c7b9ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bba4c0c7-z89pz\" (UID: \"c21dfb59-b7b0-4dfc-a842-00f203c7b9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba4c0c7-z89pz" Oct 31 19:58:17.142055 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:17.140238 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl2h6\" (UniqueName: \"kubernetes.io/projected/c21dfb59-b7b0-4dfc-a842-00f203c7b9ce-kube-api-access-jl2h6\") pod \"maas-api-revoke-sub-bba4c0c7-z89pz\" (UID: \"c21dfb59-b7b0-4dfc-a842-00f203c7b9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba4c0c7-z89pz" Oct 31 19:58:17.241493 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:17.241453 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21dfb59-b7b0-4dfc-a842-00f203c7b9ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bba4c0c7-z89pz\" (UID: \"c21dfb59-b7b0-4dfc-a842-00f203c7b9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba4c0c7-z89pz" Oct 31 19:58:17.241692 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:17.241504 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21dfb59-b7b0-4dfc-a842-00f203c7b9ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bba4c0c7-z89pz\" (UID: \"c21dfb59-b7b0-4dfc-a842-00f203c7b9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba4c0c7-z89pz" Oct 31 19:58:17.241764 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:17.241719 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl2h6\" (UniqueName: \"kubernetes.io/projected/c21dfb59-b7b0-4dfc-a842-00f203c7b9ce-kube-api-access-jl2h6\") pod \"maas-api-revoke-sub-bba4c0c7-z89pz\" (UID: \"c21dfb59-b7b0-4dfc-a842-00f203c7b9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba4c0c7-z89pz" Oct 31 19:58:17.242266 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:17.242238 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21dfb59-b7b0-4dfc-a842-00f203c7b9ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bba4c0c7-z89pz\" (UID: \"c21dfb59-b7b0-4dfc-a842-00f203c7b9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba4c0c7-z89pz" Oct 31 19:58:17.244038 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:17.244009 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21dfb59-b7b0-4dfc-a842-00f203c7b9ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bba4c0c7-z89pz\" (UID: \"c21dfb59-b7b0-4dfc-a842-00f203c7b9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba4c0c7-z89pz" Oct 31 19:58:17.255099 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:17.255068 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl2h6\" (UniqueName: \"kubernetes.io/projected/c21dfb59-b7b0-4dfc-a842-00f203c7b9ce-kube-api-access-jl2h6\") pod \"maas-api-revoke-sub-bba4c0c7-z89pz\" (UID: \"c21dfb59-b7b0-4dfc-a842-00f203c7b9ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba4c0c7-z89pz" Oct 31 19:58:17.416959 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:17.416854 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba4c0c7-z89pz" Oct 31 19:58:17.791509 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:17.791481 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bba4c0c7-z89pz"] Oct 31 19:58:18.585363 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:18.585324 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba4c0c7-z89pz" event={"ID":"c21dfb59-b7b0-4dfc-a842-00f203c7b9ce","Type":"ContainerStarted","Data":"7c27e8f35b7163da4238209bdc604f4e0d3158db105a28ba69af2faedc284f4b"} Oct 31 19:58:18.585799 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:18.585418 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba4c0c7-z89pz" event={"ID":"c21dfb59-b7b0-4dfc-a842-00f203c7b9ce","Type":"ContainerStarted","Data":"549b5a661958bc5ac432debeb6dc98199e9c5ca8dbf14d910029803d718be446"} Oct 31 19:58:18.610423 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:18.610369 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba4c0c7-z89pz" podStartSLOduration=1.610354481 podStartE2EDuration="1.610354481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:58:18.607639496 +0000 UTC m=+2332.970459870" watchObservedRunningTime="2026-10-31 19:58:18.610354481 +0000 UTC m=+2332.973174844" Oct 31 19:58:19.594814 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:19.594773 4910 generic.go:356] "Generic (PLEG): container finished" podID="c21dfb59-b7b0-4dfc-a842-00f203c7b9ce" containerID="7c27e8f35b7163da4238209bdc604f4e0d3158db105a28ba69af2faedc284f4b" exitCode=0 Oct 31 19:58:19.595246 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:19.594847 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba4c0c7-z89pz" event={"ID":"c21dfb59-b7b0-4dfc-a842-00f203c7b9ce","Type":"ContainerDied","Data":"7c27e8f35b7163da4238209bdc604f4e0d3158db105a28ba69af2faedc284f4b"} Oct 31 19:58:20.794285 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:20.794254 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba4c0c7-z89pz" Oct 31 19:58:20.822421 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:20.822386 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c21dfb59-b7b0-4dfc-a842-00f203c7b9ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21dfb59-b7b0-4dfc-a842-00f203c7b9ce-maas-api-openshift-ca\") pod \"c21dfb59-b7b0-4dfc-a842-00f203c7b9ce\" (UID: \"c21dfb59-b7b0-4dfc-a842-00f203c7b9ce\") " Oct 31 19:58:20.822596 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:20.822433 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c21dfb59-b7b0-4dfc-a842-00f203c7b9ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21dfb59-b7b0-4dfc-a842-00f203c7b9ce-maas-api-xks-ca\") pod \"c21dfb59-b7b0-4dfc-a842-00f203c7b9ce\" (UID: \"c21dfb59-b7b0-4dfc-a842-00f203c7b9ce\") " Oct 31 19:58:20.822596 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:20.822546 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c21dfb59-b7b0-4dfc-a842-00f203c7b9ce-kube-api-access-jl2h6\" (UniqueName: \"kubernetes.io/projected/c21dfb59-b7b0-4dfc-a842-00f203c7b9ce-kube-api-access-jl2h6\") pod \"c21dfb59-b7b0-4dfc-a842-00f203c7b9ce\" (UID: \"c21dfb59-b7b0-4dfc-a842-00f203c7b9ce\") " Oct 31 19:58:20.823058 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:20.823029 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c21dfb59-b7b0-4dfc-a842-00f203c7b9ce-maas-api-openshift-ca" pod "c21dfb59-b7b0-4dfc-a842-00f203c7b9ce" (UID: "c21dfb59-b7b0-4dfc-a842-00f203c7b9ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:58:20.824341 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:20.824312 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c21dfb59-b7b0-4dfc-a842-00f203c7b9ce-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:20.825274 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:20.825247 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c21dfb59-b7b0-4dfc-a842-00f203c7b9ce-kube-api-access-jl2h6" pod "c21dfb59-b7b0-4dfc-a842-00f203c7b9ce" (UID: "c21dfb59-b7b0-4dfc-a842-00f203c7b9ce"). InnerVolumeSpecName "kube-api-access-jl2h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:58:20.829644 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:20.829612 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c21dfb59-b7b0-4dfc-a842-00f203c7b9ce-maas-api-xks-ca" pod "c21dfb59-b7b0-4dfc-a842-00f203c7b9ce" (UID: "c21dfb59-b7b0-4dfc-a842-00f203c7b9ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:58:20.941991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:20.926251 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c21dfb59-b7b0-4dfc-a842-00f203c7b9ce-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:20.941991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:20.926289 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl2h6\" (UniqueName: \"kubernetes.io/projected/c21dfb59-b7b0-4dfc-a842-00f203c7b9ce-kube-api-access-jl2h6\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:21.614627 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:21.614593 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba4c0c7-z89pz" Oct 31 19:58:21.614810 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:21.614593 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bba4c0c7-z89pz" event={"ID":"c21dfb59-b7b0-4dfc-a842-00f203c7b9ce","Type":"ContainerDied","Data":"549b5a661958bc5ac432debeb6dc98199e9c5ca8dbf14d910029803d718be446"} Oct 31 19:58:21.614810 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:21.614717 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="549b5a661958bc5ac432debeb6dc98199e9c5ca8dbf14d910029803d718be446" Oct 31 19:58:22.646341 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.646299 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq"] Oct 31 19:58:22.647438 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.647410 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c21dfb59-b7b0-4dfc-a842-00f203c7b9ce" containerName="revoke-keys" Oct 31 19:58:22.647438 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.647439 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c21dfb59-b7b0-4dfc-a842-00f203c7b9ce" containerName="revoke-keys" Oct 31 19:58:22.647644 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.647626 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c21dfb59-b7b0-4dfc-a842-00f203c7b9ce" containerName="revoke-keys" Oct 31 19:58:22.652120 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.652094 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.657898 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.657874 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq"] Oct 31 19:58:22.682174 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.679798 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-26cd59-openshift-default-dockercfg-9ckxb\"" Oct 31 19:58:22.814632 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.814576 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-data\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.815034 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.815005 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-podinfo\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.815187 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.815171 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67aa01d5-e453-4154-88f3-ef247ff5949e-istiod-ca-cert\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.815302 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.815288 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-workload-socket\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.815465 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.815451 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-envoy\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.815621 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.815606 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-credential-socket\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.815778 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.815763 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfw2d\" (UniqueName: \"kubernetes.io/projected/67aa01d5-e453-4154-88f3-ef247ff5949e-kube-api-access-pfw2d\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.815928 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.815898 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-token\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.816077 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.816059 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-workload-certs\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.948438 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.940864 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfw2d\" (UniqueName: \"kubernetes.io/projected/67aa01d5-e453-4154-88f3-ef247ff5949e-kube-api-access-pfw2d\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.948438 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.940959 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-token\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.948438 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.941018 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-workload-certs\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.948438 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.941148 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-data\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.948438 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.941237 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-podinfo\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.948438 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.941265 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67aa01d5-e453-4154-88f3-ef247ff5949e-istiod-ca-cert\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.948438 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.941299 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-workload-socket\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.948438 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.941412 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-envoy\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.948438 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.941505 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-credential-socket\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.948438 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.942047 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-credential-socket\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.948438 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.942765 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-workload-certs\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.948438 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.943237 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-data\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.948438 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.945455 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-workload-socket\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.948438 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.946085 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67aa01d5-e453-4154-88f3-ef247ff5949e-istiod-ca-cert\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.952981 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.952187 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-podinfo\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.952981 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.952405 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-envoy\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.956969 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.956932 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfw2d\" (UniqueName: \"kubernetes.io/projected/67aa01d5-e453-4154-88f3-ef247ff5949e-kube-api-access-pfw2d\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.957324 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.957309 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-token\") pod \"e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:22.988877 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:22.988849 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:23.291691 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:58:23.291655 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67aa01d5_e453_4154_88f3_ef247ff5949e.slice/crio-fef68b7b4a8abc0c2acfbcd39f12c22348364c202d6ee86c4a580388713f8333 WatchSource:0}: Error finding container fef68b7b4a8abc0c2acfbcd39f12c22348364c202d6ee86c4a580388713f8333: Status 404 returned error can't find the container with id fef68b7b4a8abc0c2acfbcd39f12c22348364c202d6ee86c4a580388713f8333 Oct 31 19:58:23.292465 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:23.292441 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq"] Oct 31 19:58:23.295279 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:23.295242 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:58:23.295390 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:23.295327 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:58:23.295390 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:23.295371 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:58:23.657460 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:23.657417 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" event={"ID":"67aa01d5-e453-4154-88f3-ef247ff5949e","Type":"ContainerStarted","Data":"c232ac7fd865f3a8f4a69550980f2ebc3d30d8d83d6476d6e434c13949825ac9"} Oct 31 19:58:23.658001 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:23.657495 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" event={"ID":"67aa01d5-e453-4154-88f3-ef247ff5949e","Type":"ContainerStarted","Data":"fef68b7b4a8abc0c2acfbcd39f12c22348364c202d6ee86c4a580388713f8333"} Oct 31 19:58:23.682574 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:23.682475 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" podStartSLOduration=1.6824601590000001 podStartE2EDuration="1.682460159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:58:23.682366862 +0000 UTC m=+2338.045187234" watchObservedRunningTime="2026-10-31 19:58:23.682460159 +0000 UTC m=+2338.045280531" Oct 31 19:58:23.989868 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:23.989779 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:24.464474 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:24.464443 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2" Oct 31 19:58:25.046935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:25.045322 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:25.678751 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:25.678713 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:25.679744 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:25.679714 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:58:27.377475 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:27.377440 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-475f069f-76fdd"] Oct 31 19:58:27.382179 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:27.382157 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f069f-76fdd" Oct 31 19:58:27.393187 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:27.393159 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:58:27.394226 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:27.394200 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-475f069f-76fdd"] Oct 31 19:58:27.411460 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:27.411420 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:58:27.430385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:27.430344 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m966h\" (UniqueName: \"kubernetes.io/projected/de552fee-7ed3-497c-94e3-cc94c6c570a1-kube-api-access-m966h\") pod \"maas-api-revoke-sub-475f069f-76fdd\" (UID: \"de552fee-7ed3-497c-94e3-cc94c6c570a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f069f-76fdd" Oct 31 19:58:27.430551 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:27.430513 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de552fee-7ed3-497c-94e3-cc94c6c570a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-475f069f-76fdd\" (UID: \"de552fee-7ed3-497c-94e3-cc94c6c570a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f069f-76fdd" Oct 31 19:58:27.430712 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:27.430694 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de552fee-7ed3-497c-94e3-cc94c6c570a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-475f069f-76fdd\" (UID: \"de552fee-7ed3-497c-94e3-cc94c6c570a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f069f-76fdd" Oct 31 19:58:27.569938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:27.534083 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de552fee-7ed3-497c-94e3-cc94c6c570a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-475f069f-76fdd\" (UID: \"de552fee-7ed3-497c-94e3-cc94c6c570a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f069f-76fdd" Oct 31 19:58:27.571675 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:27.571640 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m966h\" (UniqueName: \"kubernetes.io/projected/de552fee-7ed3-497c-94e3-cc94c6c570a1-kube-api-access-m966h\") pod \"maas-api-revoke-sub-475f069f-76fdd\" (UID: \"de552fee-7ed3-497c-94e3-cc94c6c570a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f069f-76fdd" Oct 31 19:58:27.572519 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:27.572488 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de552fee-7ed3-497c-94e3-cc94c6c570a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-475f069f-76fdd\" (UID: \"de552fee-7ed3-497c-94e3-cc94c6c570a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f069f-76fdd" Oct 31 19:58:27.573314 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:27.571438 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de552fee-7ed3-497c-94e3-cc94c6c570a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-475f069f-76fdd\" (UID: \"de552fee-7ed3-497c-94e3-cc94c6c570a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f069f-76fdd" Oct 31 19:58:27.605044 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:27.605010 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de552fee-7ed3-497c-94e3-cc94c6c570a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-475f069f-76fdd\" (UID: \"de552fee-7ed3-497c-94e3-cc94c6c570a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f069f-76fdd" Oct 31 19:58:27.605649 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:27.605588 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m966h\" (UniqueName: \"kubernetes.io/projected/de552fee-7ed3-497c-94e3-cc94c6c570a1-kube-api-access-m966h\") pod \"maas-api-revoke-sub-475f069f-76fdd\" (UID: \"de552fee-7ed3-497c-94e3-cc94c6c570a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f069f-76fdd" Oct 31 19:58:27.722983 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:27.715761 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f069f-76fdd" Oct 31 19:58:28.150317 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:58:28.150285 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde552fee_7ed3_497c_94e3_cc94c6c570a1.slice/crio-5caa0502bc65dde4d154b3500d08a915133a7f563ea37181ecfca913a43a67cc WatchSource:0}: Error finding container 5caa0502bc65dde4d154b3500d08a915133a7f563ea37181ecfca913a43a67cc: Status 404 returned error can't find the container with id 5caa0502bc65dde4d154b3500d08a915133a7f563ea37181ecfca913a43a67cc Oct 31 19:58:28.151689 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:28.151663 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-475f069f-76fdd"] Oct 31 19:58:28.744646 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:28.744605 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f069f-76fdd" event={"ID":"de552fee-7ed3-497c-94e3-cc94c6c570a1","Type":"ContainerStarted","Data":"0b1a5f062be438ba74ec43777412648275a5cebc8af60b92aa1bc946bd97f5c0"} Oct 31 19:58:28.745101 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:28.744661 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f069f-76fdd" event={"ID":"de552fee-7ed3-497c-94e3-cc94c6c570a1","Type":"ContainerStarted","Data":"5caa0502bc65dde4d154b3500d08a915133a7f563ea37181ecfca913a43a67cc"} Oct 31 19:58:28.774211 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:28.774158 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f069f-76fdd" podStartSLOduration=1.774146054 podStartE2EDuration="1.774146054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:58:28.771871829 +0000 UTC m=+2343.134692214" watchObservedRunningTime="2026-10-31 19:58:28.774146054 +0000 UTC m=+2343.136966425" Oct 31 19:58:28.811338 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:28.811303 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47"] Oct 31 19:58:28.815858 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:28.815832 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" Oct 31 19:58:28.849783 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:28.849747 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47"] Oct 31 19:58:28.854448 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:28.854422 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-01afb5-serving-cert\"" Oct 31 19:58:28.912430 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:28.912390 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4s6j\" (UniqueName: \"kubernetes.io/projected/13291a84-bbac-4efb-be09-e511060c6fe9-kube-api-access-d4s6j\") pod \"maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47\" (UID: \"13291a84-bbac-4efb-be09-e511060c6fe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" Oct 31 19:58:28.912788 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:28.912763 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13291a84-bbac-4efb-be09-e511060c6fe9-metrics-certs\") pod \"maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47\" (UID: \"13291a84-bbac-4efb-be09-e511060c6fe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" Oct 31 19:58:28.912904 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:28.912886 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13291a84-bbac-4efb-be09-e511060c6fe9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47\" (UID: \"13291a84-bbac-4efb-be09-e511060c6fe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" Oct 31 19:58:28.913004 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:28.912936 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13291a84-bbac-4efb-be09-e511060c6fe9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47\" (UID: \"13291a84-bbac-4efb-be09-e511060c6fe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" Oct 31 19:58:29.014736 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:29.014675 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13291a84-bbac-4efb-be09-e511060c6fe9-metrics-certs\") pod \"maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47\" (UID: \"13291a84-bbac-4efb-be09-e511060c6fe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" Oct 31 19:58:29.015280 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:29.015253 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13291a84-bbac-4efb-be09-e511060c6fe9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47\" (UID: \"13291a84-bbac-4efb-be09-e511060c6fe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" Oct 31 19:58:29.018936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:29.015588 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13291a84-bbac-4efb-be09-e511060c6fe9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47\" (UID: \"13291a84-bbac-4efb-be09-e511060c6fe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" Oct 31 19:58:29.019274 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:29.019252 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4s6j\" (UniqueName: \"kubernetes.io/projected/13291a84-bbac-4efb-be09-e511060c6fe9-kube-api-access-d4s6j\") pod \"maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47\" (UID: \"13291a84-bbac-4efb-be09-e511060c6fe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" Oct 31 19:58:29.023998 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:58:29.015542 4910 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-serving-cert: secret "maas-api-e2e-ipp-b-01afb5-serving-cert" not found Oct 31 19:58:29.024252 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:58:29.024220 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13291a84-bbac-4efb-be09-e511060c6fe9-maas-api-tls podName:13291a84-bbac-4efb-be09-e511060c6fe9 nodeName:}" failed. No retries permitted until 2026-10-31 19:58:29.524200511 +0000 UTC m=+2343.887020862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/13291a84-bbac-4efb-be09-e511060c6fe9-maas-api-tls") pod "maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" (UID: "13291a84-bbac-4efb-be09-e511060c6fe9") : secret "maas-api-e2e-ipp-b-01afb5-serving-cert" not found Oct 31 19:58:29.024362 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:29.016717 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13291a84-bbac-4efb-be09-e511060c6fe9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47\" (UID: \"13291a84-bbac-4efb-be09-e511060c6fe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" Oct 31 19:58:29.030973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:29.028503 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13291a84-bbac-4efb-be09-e511060c6fe9-metrics-certs\") pod \"maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47\" (UID: \"13291a84-bbac-4efb-be09-e511060c6fe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" Oct 31 19:58:29.034940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:29.034896 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4s6j\" (UniqueName: \"kubernetes.io/projected/13291a84-bbac-4efb-be09-e511060c6fe9-kube-api-access-d4s6j\") pod \"maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47\" (UID: \"13291a84-bbac-4efb-be09-e511060c6fe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" Oct 31 19:58:29.535989 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:29.532040 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13291a84-bbac-4efb-be09-e511060c6fe9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47\" (UID: \"13291a84-bbac-4efb-be09-e511060c6fe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" Oct 31 19:58:29.560782 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:29.555996 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13291a84-bbac-4efb-be09-e511060c6fe9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47\" (UID: \"13291a84-bbac-4efb-be09-e511060c6fe9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" Oct 31 19:58:29.748955 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:29.744684 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" Oct 31 19:58:29.773702 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:29.773670 4910 generic.go:356] "Generic (PLEG): container finished" podID="de552fee-7ed3-497c-94e3-cc94c6c570a1" containerID="0b1a5f062be438ba74ec43777412648275a5cebc8af60b92aa1bc946bd97f5c0" exitCode=0 Oct 31 19:58:29.773994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:29.773971 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f069f-76fdd" event={"ID":"de552fee-7ed3-497c-94e3-cc94c6c570a1","Type":"ContainerDied","Data":"0b1a5f062be438ba74ec43777412648275a5cebc8af60b92aa1bc946bd97f5c0"} Oct 31 19:58:30.128506 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:30.128470 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47"] Oct 31 19:58:30.129454 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:58:30.129423 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13291a84_bbac_4efb_be09_e511060c6fe9.slice/crio-fd112ecb79976800f8f2d013d72734e0332c71e9277e1ba8d9ccface4aff3571 WatchSource:0}: Error finding container fd112ecb79976800f8f2d013d72734e0332c71e9277e1ba8d9ccface4aff3571: Status 404 returned error can't find the container with id fd112ecb79976800f8f2d013d72734e0332c71e9277e1ba8d9ccface4aff3571 Oct 31 19:58:30.130962 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:30.130938 4910 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 19:58:30.821580 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:30.820291 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" event={"ID":"13291a84-bbac-4efb-be09-e511060c6fe9","Type":"ContainerStarted","Data":"b83655f027f4f20c35f02c3c8f588d3f4102896f214d281668cf4fcc6544738b"} Oct 31 19:58:30.821580 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:30.820350 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" event={"ID":"13291a84-bbac-4efb-be09-e511060c6fe9","Type":"ContainerStarted","Data":"fd112ecb79976800f8f2d013d72734e0332c71e9277e1ba8d9ccface4aff3571"} Oct 31 19:58:30.821580 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:30.820391 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" Oct 31 19:58:30.845488 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:30.845435 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" podStartSLOduration=2.452043584 podStartE2EDuration="2.845419827s" totalImagesPullingTime="393.376243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:58:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:58:30.841813324 +0000 UTC m=+2345.204633708" watchObservedRunningTime="2026-10-31 19:58:30.845419827 +0000 UTC m=+2345.208240227" Oct 31 19:58:31.098242 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:31.098179 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f069f-76fdd" Oct 31 19:58:31.194896 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:31.194862 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de552fee-7ed3-497c-94e3-cc94c6c570a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de552fee-7ed3-497c-94e3-cc94c6c570a1-maas-api-xks-ca\") pod \"de552fee-7ed3-497c-94e3-cc94c6c570a1\" (UID: \"de552fee-7ed3-497c-94e3-cc94c6c570a1\") " Oct 31 19:58:31.195168 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:31.195150 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de552fee-7ed3-497c-94e3-cc94c6c570a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de552fee-7ed3-497c-94e3-cc94c6c570a1-maas-api-openshift-ca\") pod \"de552fee-7ed3-497c-94e3-cc94c6c570a1\" (UID: \"de552fee-7ed3-497c-94e3-cc94c6c570a1\") " Oct 31 19:58:31.195288 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:31.195275 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de552fee-7ed3-497c-94e3-cc94c6c570a1-kube-api-access-m966h\" (UniqueName: \"kubernetes.io/projected/de552fee-7ed3-497c-94e3-cc94c6c570a1-kube-api-access-m966h\") pod \"de552fee-7ed3-497c-94e3-cc94c6c570a1\" (UID: \"de552fee-7ed3-497c-94e3-cc94c6c570a1\") " Oct 31 19:58:31.195514 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:31.195486 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de552fee-7ed3-497c-94e3-cc94c6c570a1-maas-api-openshift-ca" pod "de552fee-7ed3-497c-94e3-cc94c6c570a1" (UID: "de552fee-7ed3-497c-94e3-cc94c6c570a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:58:31.195874 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:31.195855 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de552fee-7ed3-497c-94e3-cc94c6c570a1-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:31.197350 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:31.197323 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de552fee-7ed3-497c-94e3-cc94c6c570a1-maas-api-xks-ca" pod "de552fee-7ed3-497c-94e3-cc94c6c570a1" (UID: "de552fee-7ed3-497c-94e3-cc94c6c570a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:58:31.197498 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:31.197477 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de552fee-7ed3-497c-94e3-cc94c6c570a1-kube-api-access-m966h" pod "de552fee-7ed3-497c-94e3-cc94c6c570a1" (UID: "de552fee-7ed3-497c-94e3-cc94c6c570a1"). InnerVolumeSpecName "kube-api-access-m966h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:58:31.275279 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:31.275247 4910 scope.go:122] "RemoveContainer" containerID="2b36c53d49b3b90e6026df5e4950cff7ee7f0915974cf83215ae7e7570a58123" Oct 31 19:58:31.291467 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:31.291439 4910 scope.go:122] "RemoveContainer" containerID="ca655599226a631399cc6fb69d78b3a87ba9bfd7b1e5e6873d5df55405244375" Oct 31 19:58:31.298037 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:31.298009 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de552fee-7ed3-497c-94e3-cc94c6c570a1-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:31.298135 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:31.298042 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m966h\" (UniqueName: \"kubernetes.io/projected/de552fee-7ed3-497c-94e3-cc94c6c570a1-kube-api-access-m966h\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:31.301783 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:31.301766 4910 scope.go:122] "RemoveContainer" containerID="9b3c751d1209223707d9eb8ea8a94a3c045e6b33dc26a46cca8433697ae46275" Oct 31 19:58:31.832669 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:31.832637 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f069f-76fdd" Oct 31 19:58:31.833156 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:31.832646 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475f069f-76fdd" event={"ID":"de552fee-7ed3-497c-94e3-cc94c6c570a1","Type":"ContainerDied","Data":"5caa0502bc65dde4d154b3500d08a915133a7f563ea37181ecfca913a43a67cc"} Oct 31 19:58:31.833156 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:31.832768 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5caa0502bc65dde4d154b3500d08a915133a7f563ea37181ecfca913a43a67cc" Oct 31 19:58:36.839419 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:36.839388 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" Oct 31 19:58:42.731041 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:58:42.731000 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:58:50.405447 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:50.405409 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2"] Oct 31 19:58:53.072780 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:53.072741 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f"] Oct 31 19:58:53.073211 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:53.072979 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" podUID="c73a891d-f2de-4099-9a98-bdcbae51888f" containerName="istio-proxy" containerID="cri-o://cddbb54274b3fa3ecdd17a9937e1602871e13343544932ac9cbcaca01e01f56d" gracePeriod=30 Oct 31 19:58:55.409095 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:55.409045 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2" podUID="45fc8f3a-bd26-4da4-abcd-68f875727ff3" containerName="payload-pre-processing" containerID="cri-o://ebdfcc9fdbf99a21d9c08a807e242a675368f9d73ef48e1202a3d7412674dd08" gracePeriod=25 Oct 31 19:58:55.772087 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:55.772045 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2" Oct 31 19:58:55.853332 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:55.853273 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45fc8f3a-bd26-4da4-abcd-68f875727ff3-kube-api-access-js7hj\" (UniqueName: \"kubernetes.io/projected/45fc8f3a-bd26-4da4-abcd-68f875727ff3-kube-api-access-js7hj\") pod \"45fc8f3a-bd26-4da4-abcd-68f875727ff3\" (UID: \"45fc8f3a-bd26-4da4-abcd-68f875727ff3\") " Oct 31 19:58:55.853501 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:55.853410 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45fc8f3a-bd26-4da4-abcd-68f875727ff3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45fc8f3a-bd26-4da4-abcd-68f875727ff3-plugins-config-volume\") pod \"45fc8f3a-bd26-4da4-abcd-68f875727ff3\" (UID: \"45fc8f3a-bd26-4da4-abcd-68f875727ff3\") " Oct 31 19:58:55.855338 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:55.855305 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45fc8f3a-bd26-4da4-abcd-68f875727ff3-plugins-config-volume" pod "45fc8f3a-bd26-4da4-abcd-68f875727ff3" (UID: "45fc8f3a-bd26-4da4-abcd-68f875727ff3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:58:55.864488 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:55.862613 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45fc8f3a-bd26-4da4-abcd-68f875727ff3-kube-api-access-js7hj" pod "45fc8f3a-bd26-4da4-abcd-68f875727ff3" (UID: "45fc8f3a-bd26-4da4-abcd-68f875727ff3"). InnerVolumeSpecName "kube-api-access-js7hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:58:55.955829 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:55.955738 4910 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45fc8f3a-bd26-4da4-abcd-68f875727ff3-plugins-config-volume\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:55.955829 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:55.955774 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js7hj\" (UniqueName: \"kubernetes.io/projected/45fc8f3a-bd26-4da4-abcd-68f875727ff3-kube-api-access-js7hj\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:56.193204 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.193166 4910 generic.go:356] "Generic (PLEG): container finished" podID="45fc8f3a-bd26-4da4-abcd-68f875727ff3" containerID="ebdfcc9fdbf99a21d9c08a807e242a675368f9d73ef48e1202a3d7412674dd08" exitCode=0 Oct 31 19:58:56.193416 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.193219 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2" Oct 31 19:58:56.193416 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.193244 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2" event={"ID":"45fc8f3a-bd26-4da4-abcd-68f875727ff3","Type":"ContainerDied","Data":"ebdfcc9fdbf99a21d9c08a807e242a675368f9d73ef48e1202a3d7412674dd08"} Oct 31 19:58:56.193416 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.193321 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2" event={"ID":"45fc8f3a-bd26-4da4-abcd-68f875727ff3","Type":"ContainerDied","Data":"455d99c1e98bd8644fafe27c7bf57e9521a2b3d86da728dc340a29dbf4fd8a0d"} Oct 31 19:58:56.193416 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.193344 4910 scope.go:122] "RemoveContainer" containerID="ebdfcc9fdbf99a21d9c08a807e242a675368f9d73ef48e1202a3d7412674dd08" Oct 31 19:58:56.207205 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.207109 4910 scope.go:122] "RemoveContainer" containerID="ebdfcc9fdbf99a21d9c08a807e242a675368f9d73ef48e1202a3d7412674dd08" Oct 31 19:58:56.209949 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:58:56.207406 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebdfcc9fdbf99a21d9c08a807e242a675368f9d73ef48e1202a3d7412674dd08\": container with ID starting with ebdfcc9fdbf99a21d9c08a807e242a675368f9d73ef48e1202a3d7412674dd08 not found: ID does not exist" containerID="ebdfcc9fdbf99a21d9c08a807e242a675368f9d73ef48e1202a3d7412674dd08" Oct 31 19:58:56.209949 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.207442 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebdfcc9fdbf99a21d9c08a807e242a675368f9d73ef48e1202a3d7412674dd08"} err="failed to get container status \"ebdfcc9fdbf99a21d9c08a807e242a675368f9d73ef48e1202a3d7412674dd08\": rpc error: code = NotFound desc = could not find container \"ebdfcc9fdbf99a21d9c08a807e242a675368f9d73ef48e1202a3d7412674dd08\": container with ID starting with ebdfcc9fdbf99a21d9c08a807e242a675368f9d73ef48e1202a3d7412674dd08 not found: ID does not exist" Oct 31 19:58:56.227534 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.227510 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2"] Oct 31 19:58:56.234490 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.234464 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-b944c7955-85np2"] Oct 31 19:58:56.783864 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.783827 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn"] Oct 31 19:58:56.784547 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.784527 4910 cpu_manager.go:414] "containerMap: removing container" podUID="de552fee-7ed3-497c-94e3-cc94c6c570a1" containerName="revoke-keys" Oct 31 19:58:56.784547 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.784546 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de552fee-7ed3-497c-94e3-cc94c6c570a1" containerName="revoke-keys" Oct 31 19:58:56.784733 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.784578 4910 cpu_manager.go:414] "containerMap: removing container" podUID="45fc8f3a-bd26-4da4-abcd-68f875727ff3" containerName="payload-pre-processing" Oct 31 19:58:56.784733 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.784583 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45fc8f3a-bd26-4da4-abcd-68f875727ff3" containerName="payload-pre-processing" Oct 31 19:58:56.784733 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.784707 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="45fc8f3a-bd26-4da4-abcd-68f875727ff3" containerName="payload-pre-processing" Oct 31 19:58:56.784733 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.784721 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="de552fee-7ed3-497c-94e3-cc94c6c570a1" containerName="revoke-keys" Oct 31 19:58:56.790104 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.790082 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:58:56.804576 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.804551 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-cd9729\"/\"default-dockercfg-fb9l5\"" Oct 31 19:58:56.805728 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.805493 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-cd9729\"/\"kube-root-ca.crt\"" Oct 31 19:58:56.805885 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.805582 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-cd9729\"/\"openshift-service-ca.crt\"" Oct 31 19:58:56.806014 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.805681 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-cd9729\"/\"sub-test-model-cd9729-kserve-self-signed-certs\"" Oct 31 19:58:56.815264 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.815233 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn"] Oct 31 19:58:56.874533 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.874492 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-tls-certs\") pod \"sub-test-model-cd9729-kserve-8c4478c65-zw5gn\" (UID: \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:58:56.874762 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.874700 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-model-cache\") pod \"sub-test-model-cd9729-kserve-8c4478c65-zw5gn\" (UID: \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:58:56.874762 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.874748 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-dshm\") pod \"sub-test-model-cd9729-kserve-8c4478c65-zw5gn\" (UID: \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:58:56.874904 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.874840 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vkbj\" (UniqueName: \"kubernetes.io/projected/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-kube-api-access-2vkbj\") pod \"sub-test-model-cd9729-kserve-8c4478c65-zw5gn\" (UID: \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:58:56.875038 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.875021 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-home\") pod \"sub-test-model-cd9729-kserve-8c4478c65-zw5gn\" (UID: \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:58:56.875103 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.875050 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-tmp-dir\") pod \"sub-test-model-cd9729-kserve-8c4478c65-zw5gn\" (UID: \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:58:56.997946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.987398 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-tls-certs\") pod \"sub-test-model-cd9729-kserve-8c4478c65-zw5gn\" (UID: \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:58:56.997946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.987553 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-model-cache\") pod \"sub-test-model-cd9729-kserve-8c4478c65-zw5gn\" (UID: \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:58:56.997946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.987594 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-dshm\") pod \"sub-test-model-cd9729-kserve-8c4478c65-zw5gn\" (UID: \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:58:56.997946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.987642 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vkbj\" (UniqueName: \"kubernetes.io/projected/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-kube-api-access-2vkbj\") pod \"sub-test-model-cd9729-kserve-8c4478c65-zw5gn\" (UID: \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:58:56.997946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.987832 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-home\") pod \"sub-test-model-cd9729-kserve-8c4478c65-zw5gn\" (UID: \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:58:56.997946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.987874 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-tmp-dir\") pod \"sub-test-model-cd9729-kserve-8c4478c65-zw5gn\" (UID: \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:58:56.997946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.988513 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-tmp-dir\") pod \"sub-test-model-cd9729-kserve-8c4478c65-zw5gn\" (UID: \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:58:56.997946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.989787 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-model-cache\") pod \"sub-test-model-cd9729-kserve-8c4478c65-zw5gn\" (UID: \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:58:56.997946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:56.990106 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-home\") pod \"sub-test-model-cd9729-kserve-8c4478c65-zw5gn\" (UID: \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:58:57.031937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:57.016657 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-tls-certs\") pod \"sub-test-model-cd9729-kserve-8c4478c65-zw5gn\" (UID: \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:58:57.048982 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:57.047987 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-dshm\") pod \"sub-test-model-cd9729-kserve-8c4478c65-zw5gn\" (UID: \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:58:57.058935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:57.053341 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vkbj\" (UniqueName: \"kubernetes.io/projected/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-kube-api-access-2vkbj\") pod \"sub-test-model-cd9729-kserve-8c4478c65-zw5gn\" (UID: \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:58:57.116936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:57.113363 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:58:57.561003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:57.560903 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn"] Oct 31 19:58:57.758928 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:57.758878 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45fc8f3a-bd26-4da4-abcd-68f875727ff3" path="/var/lib/kubelet/pods/45fc8f3a-bd26-4da4-abcd-68f875727ff3/volumes" Oct 31 19:58:58.272534 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:58.272494 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" event={"ID":"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030","Type":"ContainerStarted","Data":"d650d1129e7d54d0c6e988d21653d0da8927eb7aeb36b9f32b8b365e14cd377f"} Oct 31 19:58:58.273019 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:58.272544 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" event={"ID":"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030","Type":"ContainerStarted","Data":"c8569bdb682689d34525e55d66342a39896dc8c7b5bf20b0f24650d7927fbefb"} Oct 31 19:58:58.274235 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:58.274212 4910 generic.go:356] "Generic (PLEG): container finished" podID="c73a891d-f2de-4099-9a98-bdcbae51888f" containerID="cddbb54274b3fa3ecdd17a9937e1602871e13343544932ac9cbcaca01e01f56d" exitCode=0 Oct 31 19:58:58.274348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:58.274268 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" event={"ID":"c73a891d-f2de-4099-9a98-bdcbae51888f","Type":"ContainerDied","Data":"cddbb54274b3fa3ecdd17a9937e1602871e13343544932ac9cbcaca01e01f56d"} Oct 31 19:58:58.295830 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:58.295787 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" podStartSLOduration=2.295772541 podStartE2EDuration="2.295772541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:58:58.292629122 +0000 UTC m=+2372.655449494" watchObservedRunningTime="2026-10-31 19:58:58.295772541 +0000 UTC m=+2372.658592913" Oct 31 19:58:58.819852 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:58.818632 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:58:58.952256 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:58.952151 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-workload-socket\") pod \"c73a891d-f2de-4099-9a98-bdcbae51888f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " Oct 31 19:58:58.952576 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:58.952267 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-workload-certs\") pod \"c73a891d-f2de-4099-9a98-bdcbae51888f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " Oct 31 19:58:58.952576 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:58.952354 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-credential-socket\") pod \"c73a891d-f2de-4099-9a98-bdcbae51888f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " Oct 31 19:58:58.952576 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:58.952418 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-data\") pod \"c73a891d-f2de-4099-9a98-bdcbae51888f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " Oct 31 19:58:58.952576 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:58.952466 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c73a891d-f2de-4099-9a98-bdcbae51888f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73a891d-f2de-4099-9a98-bdcbae51888f-istiod-ca-cert\") pod \"c73a891d-f2de-4099-9a98-bdcbae51888f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " Oct 31 19:58:58.952576 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:58.952515 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c73a891d-f2de-4099-9a98-bdcbae51888f-kube-api-access-hhdzc\" (UniqueName: \"kubernetes.io/projected/c73a891d-f2de-4099-9a98-bdcbae51888f-kube-api-access-hhdzc\") pod \"c73a891d-f2de-4099-9a98-bdcbae51888f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " Oct 31 19:58:58.952939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:58.952644 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-token\" (UniqueName: \"kubernetes.io/projected/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-token\") pod \"c73a891d-f2de-4099-9a98-bdcbae51888f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " Oct 31 19:58:58.952939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:58.952718 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-podinfo\") pod \"c73a891d-f2de-4099-9a98-bdcbae51888f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " Oct 31 19:58:58.952939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:58.952868 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-envoy\") pod \"c73a891d-f2de-4099-9a98-bdcbae51888f\" (UID: \"c73a891d-f2de-4099-9a98-bdcbae51888f\") " Oct 31 19:58:58.972517 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:58.952509 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-workload-socket" pod "c73a891d-f2de-4099-9a98-bdcbae51888f" (UID: "c73a891d-f2de-4099-9a98-bdcbae51888f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:58:58.972517 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:58.963522 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-credential-socket" pod "c73a891d-f2de-4099-9a98-bdcbae51888f" (UID: "c73a891d-f2de-4099-9a98-bdcbae51888f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:58:58.973278 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:58.973243 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-workload-certs" pod "c73a891d-f2de-4099-9a98-bdcbae51888f" (UID: "c73a891d-f2de-4099-9a98-bdcbae51888f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:58:58.975198 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:58.974997 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-data" pod "c73a891d-f2de-4099-9a98-bdcbae51888f" (UID: "c73a891d-f2de-4099-9a98-bdcbae51888f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:58:58.975198 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:58.975173 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c73a891d-f2de-4099-9a98-bdcbae51888f-istiod-ca-cert" pod "c73a891d-f2de-4099-9a98-bdcbae51888f" (UID: "c73a891d-f2de-4099-9a98-bdcbae51888f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:58:58.979933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:58.979102 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-envoy" pod "c73a891d-f2de-4099-9a98-bdcbae51888f" (UID: "c73a891d-f2de-4099-9a98-bdcbae51888f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:58:58.982939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:58.981102 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-podinfo" pod "c73a891d-f2de-4099-9a98-bdcbae51888f" (UID: "c73a891d-f2de-4099-9a98-bdcbae51888f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:58:58.985079 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:58.985051 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73a891d-f2de-4099-9a98-bdcbae51888f-kube-api-access-hhdzc" pod "c73a891d-f2de-4099-9a98-bdcbae51888f" (UID: "c73a891d-f2de-4099-9a98-bdcbae51888f"). InnerVolumeSpecName "kube-api-access-hhdzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:58:58.986298 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:58.986272 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-token" pod "c73a891d-f2de-4099-9a98-bdcbae51888f" (UID: "c73a891d-f2de-4099-9a98-bdcbae51888f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:58:59.072369 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:59.072330 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhdzc\" (UniqueName: \"kubernetes.io/projected/c73a891d-f2de-4099-9a98-bdcbae51888f-kube-api-access-hhdzc\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:59.072369 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:59.072366 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:59.072369 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:59.072376 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:59.072623 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:59.072387 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:59.072623 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:59.072395 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:59.072623 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:59.072403 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:59.072623 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:59.072411 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:59.072623 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:59.072420 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c73a891d-f2de-4099-9a98-bdcbae51888f-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:59.072623 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:59.072428 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c73a891d-f2de-4099-9a98-bdcbae51888f-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:58:59.114166 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:59.114132 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:58:59.284188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:59.284105 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" Oct 31 19:58:59.284188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:59.284144 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f" event={"ID":"c73a891d-f2de-4099-9a98-bdcbae51888f","Type":"ContainerDied","Data":"b84503ed694e13a7613cf062cb917c98ae1d91d067f480a91c27071e607c3b3d"} Oct 31 19:58:59.284732 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:59.284204 4910 scope.go:122] "RemoveContainer" containerID="cddbb54274b3fa3ecdd17a9937e1602871e13343544932ac9cbcaca01e01f56d" Oct 31 19:58:59.312405 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:59.312375 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f"] Oct 31 19:58:59.318093 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:59.318062 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6qm4f"] Oct 31 19:58:59.760033 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:58:59.759993 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c73a891d-f2de-4099-9a98-bdcbae51888f" path="/var/lib/kubelet/pods/c73a891d-f2de-4099-9a98-bdcbae51888f/volumes" Oct 31 19:59:04.182123 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:04.182083 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs"] Oct 31 19:59:04.182555 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:04.182298 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" podUID="4f470386-3ce5-46c1-9c84-68eba9905d6a" containerName="istio-proxy" containerID="cri-o://8be4abb8922c9e19636bcb8b07e453982265b9bf3a32629f4469babecb96ee5c" gracePeriod=30 Oct 31 19:59:07.113850 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:07.113761 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:59:07.126944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:07.126882 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:59:07.471042 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:07.470734 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:59:09.481342 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.481254 4910 generic.go:356] "Generic (PLEG): container finished" podID="4f470386-3ce5-46c1-9c84-68eba9905d6a" containerID="8be4abb8922c9e19636bcb8b07e453982265b9bf3a32629f4469babecb96ee5c" exitCode=0 Oct 31 19:59:09.481939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.481896 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" event={"ID":"4f470386-3ce5-46c1-9c84-68eba9905d6a","Type":"ContainerDied","Data":"8be4abb8922c9e19636bcb8b07e453982265b9bf3a32629f4469babecb96ee5c"} Oct 31 19:59:09.646682 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.646648 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:59:09.688399 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.688364 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-podinfo\") pod \"4f470386-3ce5-46c1-9c84-68eba9905d6a\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " Oct 31 19:59:09.688706 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.688685 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-workload-socket\") pod \"4f470386-3ce5-46c1-9c84-68eba9905d6a\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " Oct 31 19:59:09.688884 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.688867 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-data\") pod \"4f470386-3ce5-46c1-9c84-68eba9905d6a\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " Oct 31 19:59:09.689086 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.689067 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-credential-socket\") pod \"4f470386-3ce5-46c1-9c84-68eba9905d6a\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " Oct 31 19:59:09.689243 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.689227 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f470386-3ce5-46c1-9c84-68eba9905d6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f470386-3ce5-46c1-9c84-68eba9905d6a-istiod-ca-cert\") pod \"4f470386-3ce5-46c1-9c84-68eba9905d6a\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " Oct 31 19:59:09.689329 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.689257 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-workload-certs\") pod \"4f470386-3ce5-46c1-9c84-68eba9905d6a\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " Oct 31 19:59:09.689329 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.689315 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-envoy\") pod \"4f470386-3ce5-46c1-9c84-68eba9905d6a\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " Oct 31 19:59:09.689437 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.689376 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f470386-3ce5-46c1-9c84-68eba9905d6a-kube-api-access-zwkr5\" (UniqueName: \"kubernetes.io/projected/4f470386-3ce5-46c1-9c84-68eba9905d6a-kube-api-access-zwkr5\") pod \"4f470386-3ce5-46c1-9c84-68eba9905d6a\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " Oct 31 19:59:09.689506 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.689492 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-token\" (UniqueName: \"kubernetes.io/projected/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-token\") pod \"4f470386-3ce5-46c1-9c84-68eba9905d6a\" (UID: \"4f470386-3ce5-46c1-9c84-68eba9905d6a\") " Oct 31 19:59:09.689604 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.688943 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-workload-socket" pod "4f470386-3ce5-46c1-9c84-68eba9905d6a" (UID: "4f470386-3ce5-46c1-9c84-68eba9905d6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:59:09.689655 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.689129 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-data" pod "4f470386-3ce5-46c1-9c84-68eba9905d6a" (UID: "4f470386-3ce5-46c1-9c84-68eba9905d6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:59:09.689655 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.689311 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-credential-socket" pod "4f470386-3ce5-46c1-9c84-68eba9905d6a" (UID: "4f470386-3ce5-46c1-9c84-68eba9905d6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:59:09.689655 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.689580 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f470386-3ce5-46c1-9c84-68eba9905d6a-istiod-ca-cert" pod "4f470386-3ce5-46c1-9c84-68eba9905d6a" (UID: "4f470386-3ce5-46c1-9c84-68eba9905d6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:59:09.690276 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.690248 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f470386-3ce5-46c1-9c84-68eba9905d6a-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:09.690276 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.690277 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:09.690459 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.690292 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:09.690459 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.690306 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:09.690459 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.690307 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-workload-certs" pod "4f470386-3ce5-46c1-9c84-68eba9905d6a" (UID: "4f470386-3ce5-46c1-9c84-68eba9905d6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:59:09.692351 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.692296 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-envoy" pod "4f470386-3ce5-46c1-9c84-68eba9905d6a" (UID: "4f470386-3ce5-46c1-9c84-68eba9905d6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 19:59:09.696751 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.694064 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f470386-3ce5-46c1-9c84-68eba9905d6a-kube-api-access-zwkr5" pod "4f470386-3ce5-46c1-9c84-68eba9905d6a" (UID: "4f470386-3ce5-46c1-9c84-68eba9905d6a"). InnerVolumeSpecName "kube-api-access-zwkr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:59:09.696751 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.694160 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-token" pod "4f470386-3ce5-46c1-9c84-68eba9905d6a" (UID: "4f470386-3ce5-46c1-9c84-68eba9905d6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:59:09.697132 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.697095 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-podinfo" pod "4f470386-3ce5-46c1-9c84-68eba9905d6a" (UID: "4f470386-3ce5-46c1-9c84-68eba9905d6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 19:59:09.798461 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.798419 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwkr5\" (UniqueName: \"kubernetes.io/projected/4f470386-3ce5-46c1-9c84-68eba9905d6a-kube-api-access-zwkr5\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:09.798461 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.798463 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:09.798761 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.798478 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:09.798761 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.798493 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:09.798761 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:09.798506 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f470386-3ce5-46c1-9c84-68eba9905d6a-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:10.560935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:10.560232 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" event={"ID":"4f470386-3ce5-46c1-9c84-68eba9905d6a","Type":"ContainerDied","Data":"8c2fa99f2a54aaca7e26b4d94e7c737600303f8354fecc87c8348f52d7eb2dfb"} Oct 31 19:59:10.560935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:10.560319 4910 scope.go:122] "RemoveContainer" containerID="8be4abb8922c9e19636bcb8b07e453982265b9bf3a32629f4469babecb96ee5c" Oct 31 19:59:10.560935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:10.560491 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs" Oct 31 19:59:10.641103 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:10.639150 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs"] Oct 31 19:59:10.670940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:10.662476 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-mrdjs"] Oct 31 19:59:10.895987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:10.892112 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:10.911188 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:10.911138 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:10.953528 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:10.953486 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:59:11.075848 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.075801 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd"] Oct 31 19:59:11.077639 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.077611 4910 cpu_manager.go:414] "containerMap: removing container" podUID="4f470386-3ce5-46c1-9c84-68eba9905d6a" containerName="istio-proxy" Oct 31 19:59:11.077794 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.077781 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f470386-3ce5-46c1-9c84-68eba9905d6a" containerName="istio-proxy" Oct 31 19:59:11.077950 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.077935 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c73a891d-f2de-4099-9a98-bdcbae51888f" containerName="istio-proxy" Oct 31 19:59:11.078065 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.078053 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c73a891d-f2de-4099-9a98-bdcbae51888f" containerName="istio-proxy" Oct 31 19:59:11.078398 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.078381 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f470386-3ce5-46c1-9c84-68eba9905d6a" containerName="istio-proxy" Oct 31 19:59:11.078497 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.078485 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c73a891d-f2de-4099-9a98-bdcbae51888f" containerName="istio-proxy" Oct 31 19:59:11.112942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.097412 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 19:59:11.112942 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.103401 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-435728\"/\"default-dockercfg-7q8xn\"" Oct 31 19:59:11.200987 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.199975 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-435728\"/\"kube-root-ca.crt\"" Oct 31 19:59:11.262007 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.212239 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-435728\"/\"sub-test-model-435728-kserve-self-signed-certs\"" Oct 31 19:59:11.262007 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.220753 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-435728\"/\"openshift-service-ca.crt\"" Oct 31 19:59:11.262007 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.253642 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-dshm\") pod \"sub-test-model-435728-kserve-859458cd67-hfwgd\" (UID: \"c20b518f-fe19-47b6-88a6-aec10c91cf71\") " pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 19:59:11.262007 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.253833 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-model-cache\") pod \"sub-test-model-435728-kserve-859458cd67-hfwgd\" (UID: \"c20b518f-fe19-47b6-88a6-aec10c91cf71\") " pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 19:59:11.262007 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.253973 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-tmp-dir\") pod \"sub-test-model-435728-kserve-859458cd67-hfwgd\" (UID: \"c20b518f-fe19-47b6-88a6-aec10c91cf71\") " pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 19:59:11.262007 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.254009 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-home\") pod \"sub-test-model-435728-kserve-859458cd67-hfwgd\" (UID: \"c20b518f-fe19-47b6-88a6-aec10c91cf71\") " pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 19:59:11.262007 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.254034 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c20b518f-fe19-47b6-88a6-aec10c91cf71-tls-certs\") pod \"sub-test-model-435728-kserve-859458cd67-hfwgd\" (UID: \"c20b518f-fe19-47b6-88a6-aec10c91cf71\") " pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 19:59:11.262007 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.254116 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8crgx\" (UniqueName: \"kubernetes.io/projected/c20b518f-fe19-47b6-88a6-aec10c91cf71-kube-api-access-8crgx\") pod \"sub-test-model-435728-kserve-859458cd67-hfwgd\" (UID: \"c20b518f-fe19-47b6-88a6-aec10c91cf71\") " pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 19:59:11.303309 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.301676 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:11.355951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.327133 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd"] Oct 31 19:59:11.358592 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.357203 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-dshm\") pod \"sub-test-model-435728-kserve-859458cd67-hfwgd\" (UID: \"c20b518f-fe19-47b6-88a6-aec10c91cf71\") " pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 19:59:11.358592 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.357392 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-model-cache\") pod \"sub-test-model-435728-kserve-859458cd67-hfwgd\" (UID: \"c20b518f-fe19-47b6-88a6-aec10c91cf71\") " pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 19:59:11.358592 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.357491 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-tmp-dir\") pod \"sub-test-model-435728-kserve-859458cd67-hfwgd\" (UID: \"c20b518f-fe19-47b6-88a6-aec10c91cf71\") " pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 19:59:11.358592 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.357524 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-home\") pod \"sub-test-model-435728-kserve-859458cd67-hfwgd\" (UID: \"c20b518f-fe19-47b6-88a6-aec10c91cf71\") " pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 19:59:11.358592 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.357553 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c20b518f-fe19-47b6-88a6-aec10c91cf71-tls-certs\") pod \"sub-test-model-435728-kserve-859458cd67-hfwgd\" (UID: \"c20b518f-fe19-47b6-88a6-aec10c91cf71\") " pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 19:59:11.358592 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.357643 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8crgx\" (UniqueName: \"kubernetes.io/projected/c20b518f-fe19-47b6-88a6-aec10c91cf71-kube-api-access-8crgx\") pod \"sub-test-model-435728-kserve-859458cd67-hfwgd\" (UID: \"c20b518f-fe19-47b6-88a6-aec10c91cf71\") " pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 19:59:11.380931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.359407 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-tmp-dir\") pod \"sub-test-model-435728-kserve-859458cd67-hfwgd\" (UID: \"c20b518f-fe19-47b6-88a6-aec10c91cf71\") " pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 19:59:11.380931 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.359673 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-model-cache\") pod \"sub-test-model-435728-kserve-859458cd67-hfwgd\" (UID: \"c20b518f-fe19-47b6-88a6-aec10c91cf71\") " pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 19:59:11.395934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.384663 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-home\") pod \"sub-test-model-435728-kserve-859458cd67-hfwgd\" (UID: \"c20b518f-fe19-47b6-88a6-aec10c91cf71\") " pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 19:59:11.395934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.385454 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:59:11.416793 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.400699 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:11.496131 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.491633 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c20b518f-fe19-47b6-88a6-aec10c91cf71-tls-certs\") pod \"sub-test-model-435728-kserve-859458cd67-hfwgd\" (UID: \"c20b518f-fe19-47b6-88a6-aec10c91cf71\") " pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 19:59:11.522980 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.508031 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:11.522980 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.509666 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" podUID="8fa2f1d6-abaa-4c73-8fdc-b91691bfb030" containerName="main" probeResult="failure" output="Get \"https://10.132.0.243:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:11.563358 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.555806 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8crgx\" (UniqueName: \"kubernetes.io/projected/c20b518f-fe19-47b6-88a6-aec10c91cf71-kube-api-access-8crgx\") pod \"sub-test-model-435728-kserve-859458cd67-hfwgd\" (UID: \"c20b518f-fe19-47b6-88a6-aec10c91cf71\") " pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 19:59:11.618978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.605349 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-dshm\") pod \"sub-test-model-435728-kserve-859458cd67-hfwgd\" (UID: \"c20b518f-fe19-47b6-88a6-aec10c91cf71\") " pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 19:59:11.618978 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.616704 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:11.719936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.701233 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 19:59:11.743935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.723996 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 19:59:11.771999 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.771236 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:11.810972 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.810190 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:11.920937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.900510 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f470386-3ce5-46c1-9c84-68eba9905d6a" path="/var/lib/kubelet/pods/4f470386-3ce5-46c1-9c84-68eba9905d6a/volumes" Oct 31 19:59:11.968937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.921933 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:11.968937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.925585 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:11.968937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.938950 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:59:11.968937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.939031 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:59:11.994062 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.989385 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:11.994062 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.990451 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 19:59:11.994062 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:11.990709 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:59:12.067941 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:12.067731 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:12.089946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:12.069000 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:12.338676 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:12.305107 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:59:12.338676 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:12.306492 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:59:12.352840 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:12.351046 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded" Oct 31 19:59:12.475349 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:12.475305 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:12.528936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:12.500809 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:59:12.528936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:12.511973 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 19:59:12.528936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:12.514622 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:59:12.571785 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:12.568976 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:12.571785 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:12.569570 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:59:12.584996 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:12.573202 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" podUID="8fa2f1d6-abaa-4c73-8fdc-b91691bfb030" containerName="main" probeResult="failure" output="Get \"https://10.132.0.243:8000/ready\": context deadline exceeded" Oct 31 19:59:12.584996 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:12.573527 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:59:12.605900 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:12.605816 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 19:59:12.665310 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:12.665269 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:12.666196 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:12.666175 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:59:12.677603 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:12.677570 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 19:59:12.677894 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:12.677853 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" podUID="ceb11f75-3d05-4e9c-8db7-caa224621cad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:12.734408 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:12.734350 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:12.740969 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:12.740943 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:59:12.783938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:12.772000 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:12.810938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:12.784793 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:59:12.854939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:12.812171 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:12.863419 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:12.819724 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 19:59:12.863722 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:12.863698 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:59:13.042097 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:13.042038 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:13.087959 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:13.053058 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 19:59:13.107900 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:13.107847 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:13.122110 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:13.122015 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:13.133036 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:13.132990 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:59:13.266582 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:13.266533 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:13.475970 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:13.473313 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 19:59:13.476698 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:13.476663 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 19:59:13.485444 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:13.483784 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 19:59:13.485444 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:59:13.481899 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:59:13.485444 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:13.484755 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 19:59:13.489212 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:13.488770 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 19:59:13.499878 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:13.498409 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 19:59:13.504574 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:13.504542 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:13.529005 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:13.528978 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 19:59:13.538929 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:13.531770 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 19:59:13.539797 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:13.538001 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:13.540219 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:13.539930 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 19:59:13.559039 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:13.559013 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 19:59:13.596695 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:13.595221 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 19:59:13.852113 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:13.828668 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd"] Oct 31 19:59:14.485927 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:14.485873 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" event={"ID":"c20b518f-fe19-47b6-88a6-aec10c91cf71","Type":"ContainerStarted","Data":"e21998d7c0ef6baefefd073f3bcf21172c6058f0b104c9e22f806771ff24c0f7"} Oct 31 19:59:14.486139 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:14.485942 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" event={"ID":"c20b518f-fe19-47b6-88a6-aec10c91cf71","Type":"ContainerStarted","Data":"2eae7c3a46310538983d181700cf4b706044fab3573b2e9aa2d1f298a3256b82"} Oct 31 19:59:14.511665 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:14.511607 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" podStartSLOduration=3.511592583 podStartE2EDuration="3.511592583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:14.506117615 +0000 UTC m=+2388.868937984" watchObservedRunningTime="2026-10-31 19:59:14.511592583 +0000 UTC m=+2388.874412956" Oct 31 19:59:14.701766 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:14.701666 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 19:59:15.564949 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:15.563992 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-096eeffc-4xmj8"] Oct 31 19:59:15.570069 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:15.570032 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096eeffc-4xmj8" Oct 31 19:59:15.576589 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:15.576541 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-096eeffc-4xmj8"] Oct 31 19:59:15.596342 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:15.596303 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:59:15.596472 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:15.596391 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:59:15.621695 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:15.619228 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/850fb91b-2e49-4d05-9e1f-23571e4ca2ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-096eeffc-4xmj8\" (UID: \"850fb91b-2e49-4d05-9e1f-23571e4ca2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096eeffc-4xmj8" Oct 31 19:59:15.621695 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:15.619412 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mzrp\" (UniqueName: \"kubernetes.io/projected/850fb91b-2e49-4d05-9e1f-23571e4ca2ec-kube-api-access-9mzrp\") pod \"maas-api-revoke-sub-096eeffc-4xmj8\" (UID: \"850fb91b-2e49-4d05-9e1f-23571e4ca2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096eeffc-4xmj8" Oct 31 19:59:15.621695 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:15.619537 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/850fb91b-2e49-4d05-9e1f-23571e4ca2ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-096eeffc-4xmj8\" (UID: \"850fb91b-2e49-4d05-9e1f-23571e4ca2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096eeffc-4xmj8" Oct 31 19:59:15.721311 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:15.721276 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mzrp\" (UniqueName: \"kubernetes.io/projected/850fb91b-2e49-4d05-9e1f-23571e4ca2ec-kube-api-access-9mzrp\") pod \"maas-api-revoke-sub-096eeffc-4xmj8\" (UID: \"850fb91b-2e49-4d05-9e1f-23571e4ca2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096eeffc-4xmj8" Oct 31 19:59:15.721780 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:15.721407 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/850fb91b-2e49-4d05-9e1f-23571e4ca2ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-096eeffc-4xmj8\" (UID: \"850fb91b-2e49-4d05-9e1f-23571e4ca2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096eeffc-4xmj8" Oct 31 19:59:15.721780 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:15.721560 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/850fb91b-2e49-4d05-9e1f-23571e4ca2ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-096eeffc-4xmj8\" (UID: \"850fb91b-2e49-4d05-9e1f-23571e4ca2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096eeffc-4xmj8" Oct 31 19:59:15.722327 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:15.722302 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/850fb91b-2e49-4d05-9e1f-23571e4ca2ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-096eeffc-4xmj8\" (UID: \"850fb91b-2e49-4d05-9e1f-23571e4ca2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096eeffc-4xmj8" Oct 31 19:59:15.733409 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:15.733379 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/850fb91b-2e49-4d05-9e1f-23571e4ca2ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-096eeffc-4xmj8\" (UID: \"850fb91b-2e49-4d05-9e1f-23571e4ca2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096eeffc-4xmj8" Oct 31 19:59:15.734734 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:15.734699 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mzrp\" (UniqueName: \"kubernetes.io/projected/850fb91b-2e49-4d05-9e1f-23571e4ca2ec-kube-api-access-9mzrp\") pod \"maas-api-revoke-sub-096eeffc-4xmj8\" (UID: \"850fb91b-2e49-4d05-9e1f-23571e4ca2ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-096eeffc-4xmj8" Oct 31 19:59:15.892447 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:15.892368 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096eeffc-4xmj8" Oct 31 19:59:16.198278 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:59:16.198236 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod850fb91b_2e49_4d05_9e1f_23571e4ca2ec.slice/crio-1f308b0733ec553dda55d73f1c01d65d7743f017ed27ee3a16f90e1227fd65c9 WatchSource:0}: Error finding container 1f308b0733ec553dda55d73f1c01d65d7743f017ed27ee3a16f90e1227fd65c9: Status 404 returned error can't find the container with id 1f308b0733ec553dda55d73f1c01d65d7743f017ed27ee3a16f90e1227fd65c9 Oct 31 19:59:16.200233 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:16.200178 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-096eeffc-4xmj8"] Oct 31 19:59:16.523761 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:16.523672 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096eeffc-4xmj8" event={"ID":"850fb91b-2e49-4d05-9e1f-23571e4ca2ec","Type":"ContainerStarted","Data":"1f308b0733ec553dda55d73f1c01d65d7743f017ed27ee3a16f90e1227fd65c9"} Oct 31 19:59:17.537044 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:17.536994 4910 generic.go:356] "Generic (PLEG): container finished" podID="850fb91b-2e49-4d05-9e1f-23571e4ca2ec" containerID="30f8a31b226e258d422416032273e363eef75e9a5b2680dffb5e1a617de93f59" exitCode=0 Oct 31 19:59:17.537674 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:17.537285 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096eeffc-4xmj8" event={"ID":"850fb91b-2e49-4d05-9e1f-23571e4ca2ec","Type":"ContainerDied","Data":"30f8a31b226e258d422416032273e363eef75e9a5b2680dffb5e1a617de93f59"} Oct 31 19:59:18.750198 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:18.750166 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096eeffc-4xmj8" Oct 31 19:59:18.902475 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:18.902437 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/850fb91b-2e49-4d05-9e1f-23571e4ca2ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/850fb91b-2e49-4d05-9e1f-23571e4ca2ec-maas-api-openshift-ca\") pod \"850fb91b-2e49-4d05-9e1f-23571e4ca2ec\" (UID: \"850fb91b-2e49-4d05-9e1f-23571e4ca2ec\") " Oct 31 19:59:18.902475 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:18.902477 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/850fb91b-2e49-4d05-9e1f-23571e4ca2ec-kube-api-access-9mzrp\" (UniqueName: \"kubernetes.io/projected/850fb91b-2e49-4d05-9e1f-23571e4ca2ec-kube-api-access-9mzrp\") pod \"850fb91b-2e49-4d05-9e1f-23571e4ca2ec\" (UID: \"850fb91b-2e49-4d05-9e1f-23571e4ca2ec\") " Oct 31 19:59:18.902739 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:18.902610 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/850fb91b-2e49-4d05-9e1f-23571e4ca2ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/850fb91b-2e49-4d05-9e1f-23571e4ca2ec-maas-api-xks-ca\") pod \"850fb91b-2e49-4d05-9e1f-23571e4ca2ec\" (UID: \"850fb91b-2e49-4d05-9e1f-23571e4ca2ec\") " Oct 31 19:59:18.902890 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:18.902860 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/850fb91b-2e49-4d05-9e1f-23571e4ca2ec-maas-api-openshift-ca" pod "850fb91b-2e49-4d05-9e1f-23571e4ca2ec" (UID: "850fb91b-2e49-4d05-9e1f-23571e4ca2ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:59:18.903459 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:18.903432 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/850fb91b-2e49-4d05-9e1f-23571e4ca2ec-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:18.904827 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:18.904802 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850fb91b-2e49-4d05-9e1f-23571e4ca2ec-kube-api-access-9mzrp" pod "850fb91b-2e49-4d05-9e1f-23571e4ca2ec" (UID: "850fb91b-2e49-4d05-9e1f-23571e4ca2ec"). InnerVolumeSpecName "kube-api-access-9mzrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:59:18.904963 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:18.904850 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/850fb91b-2e49-4d05-9e1f-23571e4ca2ec-maas-api-xks-ca" pod "850fb91b-2e49-4d05-9e1f-23571e4ca2ec" (UID: "850fb91b-2e49-4d05-9e1f-23571e4ca2ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:59:19.004962 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.004889 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mzrp\" (UniqueName: \"kubernetes.io/projected/850fb91b-2e49-4d05-9e1f-23571e4ca2ec-kube-api-access-9mzrp\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:19.004962 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.004963 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/850fb91b-2e49-4d05-9e1f-23571e4ca2ec-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:19.244819 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.244726 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd"] Oct 31 19:59:19.245544 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.245522 4910 cpu_manager.go:414] "containerMap: removing container" podUID="850fb91b-2e49-4d05-9e1f-23571e4ca2ec" containerName="revoke-keys" Oct 31 19:59:19.245544 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.245544 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="850fb91b-2e49-4d05-9e1f-23571e4ca2ec" containerName="revoke-keys" Oct 31 19:59:19.245760 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.245747 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="850fb91b-2e49-4d05-9e1f-23571e4ca2ec" containerName="revoke-keys" Oct 31 19:59:19.249838 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.249807 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.257184 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.257166 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-e1d060-openshift-default-dockercfg-w5qst\"" Oct 31 19:59:19.268332 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.268304 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd"] Oct 31 19:59:19.334348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.334313 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-credential-socket\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.334348 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.334363 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-workload-certs\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.334568 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.334512 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-envoy\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.334568 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.334554 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-workload-socket\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.334666 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.334579 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-podinfo\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.334726 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.334699 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9zd7\" (UniqueName: \"kubernetes.io/projected/e898f0b3-1c73-44d5-bc56-1d2c5673d516-kube-api-access-q9zd7\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.334885 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.334849 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istiod-ca-cert\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.335039 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.335017 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-token\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.335085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.335070 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-data\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.436320 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.436276 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-credential-socket\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.436320 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.436319 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-workload-certs\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.436568 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.436405 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-envoy\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.436568 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.436437 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-workload-socket\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.436568 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.436474 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-podinfo\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.436568 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.436513 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9zd7\" (UniqueName: \"kubernetes.io/projected/e898f0b3-1c73-44d5-bc56-1d2c5673d516-kube-api-access-q9zd7\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.436784 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.436645 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istiod-ca-cert\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.436784 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.436748 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-credential-socket\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.436952 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.436780 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-workload-certs\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.436952 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.436750 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-token\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.436952 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.436851 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-data\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.437198 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.437163 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-workload-socket\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.437277 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.437260 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-data\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.437633 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.437617 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istiod-ca-cert\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.439540 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.439512 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-envoy\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.439774 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.439753 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-podinfo\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.450047 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.450017 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-token\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.450169 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.450070 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9zd7\" (UniqueName: \"kubernetes.io/projected/e898f0b3-1c73-44d5-bc56-1d2c5673d516-kube-api-access-q9zd7\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.563496 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.563460 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096eeffc-4xmj8" event={"ID":"850fb91b-2e49-4d05-9e1f-23571e4ca2ec","Type":"ContainerDied","Data":"1f308b0733ec553dda55d73f1c01d65d7743f017ed27ee3a16f90e1227fd65c9"} Oct 31 19:59:19.563652 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.563500 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-096eeffc-4xmj8" Oct 31 19:59:19.563652 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.563519 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f308b0733ec553dda55d73f1c01d65d7743f017ed27ee3a16f90e1227fd65c9" Oct 31 19:59:19.571352 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.571329 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:19.870196 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.870163 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd"] Oct 31 19:59:19.871276 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:59:19.871250 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode898f0b3_1c73_44d5_bc56_1d2c5673d516.slice/crio-11f6280707241f19fff2092eee0ae9b557560cdabc5869bfabea8103be45c19a WatchSource:0}: Error finding container 11f6280707241f19fff2092eee0ae9b557560cdabc5869bfabea8103be45c19a: Status 404 returned error can't find the container with id 11f6280707241f19fff2092eee0ae9b557560cdabc5869bfabea8103be45c19a Oct 31 19:59:19.874251 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.874215 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:59:19.874332 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.874316 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:59:19.874396 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:19.874347 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:59:20.575614 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:20.575575 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" event={"ID":"e898f0b3-1c73-44d5-bc56-1d2c5673d516","Type":"ContainerStarted","Data":"138e83d447d96ef6fee84b68ec56c175b342faa1e7eed49813a9c169a03e1cf2"} Oct 31 19:59:20.575803 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:20.575648 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" event={"ID":"e898f0b3-1c73-44d5-bc56-1d2c5673d516","Type":"ContainerStarted","Data":"11f6280707241f19fff2092eee0ae9b557560cdabc5869bfabea8103be45c19a"} Oct 31 19:59:20.602761 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:20.602711 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" podStartSLOduration=1.6026976560000001 podStartE2EDuration="1.602697656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:20.599771493 +0000 UTC m=+2394.962591864" watchObservedRunningTime="2026-10-31 19:59:20.602697656 +0000 UTC m=+2394.965518028" Oct 31 19:59:21.194106 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.194063 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l"] Oct 31 19:59:21.198544 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.198519 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.208885 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.208851 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-01afb5-openshift-default-dockercfg-czt6t\"" Oct 31 19:59:21.212484 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.212452 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l"] Oct 31 19:59:21.385683 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.385641 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-workload-certs\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.385683 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.385684 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-workload-socket\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.386003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.385700 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-data\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.386003 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.385862 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-podinfo\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.386091 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.386062 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-credential-socket\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.386148 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.386137 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-envoy\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.386273 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.386256 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm9xb\" (UniqueName: \"kubernetes.io/projected/d75a25fa-2c40-4489-afda-aee9c5ca3610-kube-api-access-zm9xb\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.386328 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.386310 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-token\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.386435 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.386416 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75a25fa-2c40-4489-afda-aee9c5ca3610-istiod-ca-cert\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.488434 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.488337 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75a25fa-2c40-4489-afda-aee9c5ca3610-istiod-ca-cert\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.488434 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.488422 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-workload-certs\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.488673 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.488449 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-workload-socket\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.488673 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.488471 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-data\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.488673 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.488540 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-podinfo\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.488673 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.488657 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-credential-socket\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.488961 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.488756 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-envoy\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.488961 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.488836 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm9xb\" (UniqueName: \"kubernetes.io/projected/d75a25fa-2c40-4489-afda-aee9c5ca3610-kube-api-access-zm9xb\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.488961 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.488892 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-token\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.489137 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.489085 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75a25fa-2c40-4489-afda-aee9c5ca3610-istiod-ca-cert\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.489711 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.489685 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-data\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.489842 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.489774 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-credential-socket\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.489966 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.489944 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-workload-certs\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.490263 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.490243 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-workload-socket\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.491986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.491960 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-podinfo\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.492099 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.492009 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-envoy\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.501799 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.501774 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-token\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.502238 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.502218 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm9xb\" (UniqueName: \"kubernetes.io/projected/d75a25fa-2c40-4489-afda-aee9c5ca3610-kube-api-access-zm9xb\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.514888 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.514866 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:21.573331 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.571643 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:21.575017 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.574065 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 31 19:59:21.575017 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.574136 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" podUID="e898f0b3-1c73-44d5-bc56-1d2c5673d516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 31 19:59:21.704334 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.703408 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 19:59:21.774029 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.772114 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 19:59:21.862925 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.862886 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l"] Oct 31 19:59:21.869092 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:59:21.867292 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75a25fa_2c40_4489_afda_aee9c5ca3610.slice/crio-de4562e3982352fdd27d284620ecd11e11822e222d24a7173da67b56f045e0c5 WatchSource:0}: Error finding container de4562e3982352fdd27d284620ecd11e11822e222d24a7173da67b56f045e0c5: Status 404 returned error can't find the container with id de4562e3982352fdd27d284620ecd11e11822e222d24a7173da67b56f045e0c5 Oct 31 19:59:21.872289 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.872232 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:59:21.872432 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.872411 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:59:21.872528 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:21.872510 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:59:22.573930 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:22.573378 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 31 19:59:22.573930 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:22.573451 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" podUID="e898f0b3-1c73-44d5-bc56-1d2c5673d516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 31 19:59:22.596319 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:22.595626 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" event={"ID":"d75a25fa-2c40-4489-afda-aee9c5ca3610","Type":"ContainerStarted","Data":"bd15d662eefd3bda09440c08b264c658442767ff658fc00d528b09ab00d4e3b9"} Oct 31 19:59:22.596319 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:22.595721 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" event={"ID":"d75a25fa-2c40-4489-afda-aee9c5ca3610","Type":"ContainerStarted","Data":"de4562e3982352fdd27d284620ecd11e11822e222d24a7173da67b56f045e0c5"} Oct 31 19:59:22.608448 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:22.608412 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 19:59:22.656798 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:22.656693 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" podStartSLOduration=1.656673654 podStartE2EDuration="1.656673654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:22.624397015 +0000 UTC m=+2396.987217407" watchObservedRunningTime="2026-10-31 19:59:22.656673654 +0000 UTC m=+2397.019494026" Oct 31 19:59:23.516222 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:23.516154 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:23.517416 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:23.517392 4910 patch_prober.go:29] interesting pod/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 19:59:23.517483 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:23.517439 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" podUID="d75a25fa-2c40-4489-afda-aee9c5ca3610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 19:59:23.577936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:23.576896 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 31 19:59:23.577936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:23.576984 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" podUID="e898f0b3-1c73-44d5-bc56-1d2c5673d516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 31 19:59:24.515674 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:24.515637 4910 patch_prober.go:29] interesting pod/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 19:59:24.515850 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:24.515692 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" podUID="d75a25fa-2c40-4489-afda-aee9c5ca3610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 19:59:24.575937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:24.575321 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 31 19:59:24.575937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:24.575384 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" podUID="e898f0b3-1c73-44d5-bc56-1d2c5673d516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 31 19:59:25.567938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:25.562845 4910 patch_prober.go:29] interesting pod/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 19:59:25.567938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:25.562924 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" podUID="d75a25fa-2c40-4489-afda-aee9c5ca3610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 19:59:25.579933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:25.579613 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 31 19:59:25.579933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:25.579672 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" podUID="e898f0b3-1c73-44d5-bc56-1d2c5673d516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 31 19:59:26.340935 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:26.322184 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-365b4309-n4dmb"] Oct 31 19:59:26.398924 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:26.398881 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-365b4309-n4dmb"] Oct 31 19:59:26.399143 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:26.399013 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-365b4309-n4dmb" Oct 31 19:59:26.405164 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:26.405136 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:59:26.405337 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:26.405165 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:59:26.485985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:26.483003 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bf64223-cc5f-4a12-99b2-af2b1248bdbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-365b4309-n4dmb\" (UID: \"4bf64223-cc5f-4a12-99b2-af2b1248bdbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-365b4309-n4dmb" Oct 31 19:59:26.485985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:26.483117 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llhl6\" (UniqueName: \"kubernetes.io/projected/4bf64223-cc5f-4a12-99b2-af2b1248bdbe-kube-api-access-llhl6\") pod \"maas-api-revoke-sub-365b4309-n4dmb\" (UID: \"4bf64223-cc5f-4a12-99b2-af2b1248bdbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-365b4309-n4dmb" Oct 31 19:59:26.485985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:26.483324 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bf64223-cc5f-4a12-99b2-af2b1248bdbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-365b4309-n4dmb\" (UID: \"4bf64223-cc5f-4a12-99b2-af2b1248bdbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-365b4309-n4dmb" Oct 31 19:59:26.530932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:26.515993 4910 patch_prober.go:29] interesting pod/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 19:59:26.530932 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:26.516065 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" podUID="d75a25fa-2c40-4489-afda-aee9c5ca3610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 19:59:26.581181 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:26.577106 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 31 19:59:26.581181 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:26.577183 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" podUID="e898f0b3-1c73-44d5-bc56-1d2c5673d516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 31 19:59:26.590682 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:26.589864 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llhl6\" (UniqueName: \"kubernetes.io/projected/4bf64223-cc5f-4a12-99b2-af2b1248bdbe-kube-api-access-llhl6\") pod \"maas-api-revoke-sub-365b4309-n4dmb\" (UID: \"4bf64223-cc5f-4a12-99b2-af2b1248bdbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-365b4309-n4dmb" Oct 31 19:59:26.590682 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:26.590107 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bf64223-cc5f-4a12-99b2-af2b1248bdbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-365b4309-n4dmb\" (UID: \"4bf64223-cc5f-4a12-99b2-af2b1248bdbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-365b4309-n4dmb" Oct 31 19:59:26.590682 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:26.590258 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bf64223-cc5f-4a12-99b2-af2b1248bdbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-365b4309-n4dmb\" (UID: \"4bf64223-cc5f-4a12-99b2-af2b1248bdbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-365b4309-n4dmb" Oct 31 19:59:26.592422 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:26.592361 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bf64223-cc5f-4a12-99b2-af2b1248bdbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-365b4309-n4dmb\" (UID: \"4bf64223-cc5f-4a12-99b2-af2b1248bdbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-365b4309-n4dmb" Oct 31 19:59:26.603842 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:26.593923 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bf64223-cc5f-4a12-99b2-af2b1248bdbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-365b4309-n4dmb\" (UID: \"4bf64223-cc5f-4a12-99b2-af2b1248bdbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-365b4309-n4dmb" Oct 31 19:59:26.659109 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:26.642158 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llhl6\" (UniqueName: \"kubernetes.io/projected/4bf64223-cc5f-4a12-99b2-af2b1248bdbe-kube-api-access-llhl6\") pod \"maas-api-revoke-sub-365b4309-n4dmb\" (UID: \"4bf64223-cc5f-4a12-99b2-af2b1248bdbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-365b4309-n4dmb" Oct 31 19:59:26.706734 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:26.706700 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 19:59:26.751938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:26.732651 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 19:59:26.751938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:26.750904 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-365b4309-n4dmb" Oct 31 19:59:27.581937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:27.569534 4910 patch_prober.go:29] interesting pod/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 19:59:27.581937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:27.569595 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" podUID="d75a25fa-2c40-4489-afda-aee9c5ca3610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 19:59:27.581937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:27.572752 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body= Oct 31 19:59:27.581937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:27.572798 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" podUID="e898f0b3-1c73-44d5-bc56-1d2c5673d516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" Oct 31 19:59:27.713649 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:59:27.713604 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf64223_cc5f_4a12_99b2_af2b1248bdbe.slice/crio-0de0c01d253ea9021d80bd3710d88b5bf858586aa7d4b54edafc98a38823cf06 WatchSource:0}: Error finding container 0de0c01d253ea9021d80bd3710d88b5bf858586aa7d4b54edafc98a38823cf06: Status 404 returned error can't find the container with id 0de0c01d253ea9021d80bd3710d88b5bf858586aa7d4b54edafc98a38823cf06 Oct 31 19:59:27.715584 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:27.715095 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-365b4309-n4dmb"] Oct 31 19:59:27.944946 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:27.942863 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-365b4309-n4dmb" event={"ID":"4bf64223-cc5f-4a12-99b2-af2b1248bdbe","Type":"ContainerStarted","Data":"0de0c01d253ea9021d80bd3710d88b5bf858586aa7d4b54edafc98a38823cf06"} Oct 31 19:59:28.523934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:28.518231 4910 patch_prober.go:29] interesting pod/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 19:59:28.523934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:28.518291 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" podUID="d75a25fa-2c40-4489-afda-aee9c5ca3610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 19:59:28.758043 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:28.699238 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:28.758043 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:28.701249 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:28.758043 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:28.710253 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 19:59:29.198889 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:29.198397 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-365b4309-n4dmb" event={"ID":"4bf64223-cc5f-4a12-99b2-af2b1248bdbe","Type":"ContainerStarted","Data":"8150b25104a272dcc02b0c705ba81a15558e80934d66731684cb5b3a303cf963"} Oct 31 19:59:29.585136 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:29.585104 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:29.586427 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:29.586404 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:29.586934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:29.586902 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 19:59:29.616343 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:29.616299 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-365b4309-n4dmb" podStartSLOduration=4.616286511 podStartE2EDuration="4.616286511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:29.233619289 +0000 UTC m=+2403.596439662" watchObservedRunningTime="2026-10-31 19:59:29.616286511 +0000 UTC m=+2403.979106882" Oct 31 19:59:30.179489 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:30.179456 4910 generic.go:356] "Generic (PLEG): container finished" podID="4bf64223-cc5f-4a12-99b2-af2b1248bdbe" containerID="8150b25104a272dcc02b0c705ba81a15558e80934d66731684cb5b3a303cf963" exitCode=0 Oct 31 19:59:30.179957 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:30.179539 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-365b4309-n4dmb" event={"ID":"4bf64223-cc5f-4a12-99b2-af2b1248bdbe","Type":"ContainerDied","Data":"8150b25104a272dcc02b0c705ba81a15558e80934d66731684cb5b3a303cf963"} Oct 31 19:59:31.367890 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:31.367855 4910 scope.go:122] "RemoveContainer" containerID="43219138eea25a83520457e9499fe98188516b3ab1bffceec0704b56ec17785b" Oct 31 19:59:31.463206 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:31.463143 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-365b4309-n4dmb" Oct 31 19:59:31.516633 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:31.516596 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bf64223-cc5f-4a12-99b2-af2b1248bdbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bf64223-cc5f-4a12-99b2-af2b1248bdbe-maas-api-openshift-ca\") pod \"4bf64223-cc5f-4a12-99b2-af2b1248bdbe\" (UID: \"4bf64223-cc5f-4a12-99b2-af2b1248bdbe\") " Oct 31 19:59:31.516825 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:31.516697 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bf64223-cc5f-4a12-99b2-af2b1248bdbe-kube-api-access-llhl6\" (UniqueName: \"kubernetes.io/projected/4bf64223-cc5f-4a12-99b2-af2b1248bdbe-kube-api-access-llhl6\") pod \"4bf64223-cc5f-4a12-99b2-af2b1248bdbe\" (UID: \"4bf64223-cc5f-4a12-99b2-af2b1248bdbe\") " Oct 31 19:59:31.516991 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:31.516968 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bf64223-cc5f-4a12-99b2-af2b1248bdbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bf64223-cc5f-4a12-99b2-af2b1248bdbe-maas-api-xks-ca\") pod \"4bf64223-cc5f-4a12-99b2-af2b1248bdbe\" (UID: \"4bf64223-cc5f-4a12-99b2-af2b1248bdbe\") " Oct 31 19:59:31.517074 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:31.517015 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bf64223-cc5f-4a12-99b2-af2b1248bdbe-maas-api-openshift-ca" pod "4bf64223-cc5f-4a12-99b2-af2b1248bdbe" (UID: "4bf64223-cc5f-4a12-99b2-af2b1248bdbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:59:31.517851 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:31.517822 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bf64223-cc5f-4a12-99b2-af2b1248bdbe-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:31.561954 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:31.539353 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bf64223-cc5f-4a12-99b2-af2b1248bdbe-maas-api-xks-ca" pod "4bf64223-cc5f-4a12-99b2-af2b1248bdbe" (UID: "4bf64223-cc5f-4a12-99b2-af2b1248bdbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:59:31.571109 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:31.571071 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf64223-cc5f-4a12-99b2-af2b1248bdbe-kube-api-access-llhl6" pod "4bf64223-cc5f-4a12-99b2-af2b1248bdbe" (UID: "4bf64223-cc5f-4a12-99b2-af2b1248bdbe"). InnerVolumeSpecName "kube-api-access-llhl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:59:31.623558 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:31.619459 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bf64223-cc5f-4a12-99b2-af2b1248bdbe-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:31.624779 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:31.624756 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llhl6\" (UniqueName: \"kubernetes.io/projected/4bf64223-cc5f-4a12-99b2-af2b1248bdbe-kube-api-access-llhl6\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:32.196401 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.196317 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-365b4309-n4dmb" event={"ID":"4bf64223-cc5f-4a12-99b2-af2b1248bdbe","Type":"ContainerDied","Data":"0de0c01d253ea9021d80bd3710d88b5bf858586aa7d4b54edafc98a38823cf06"} Oct 31 19:59:32.196401 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.196372 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0de0c01d253ea9021d80bd3710d88b5bf858586aa7d4b54edafc98a38823cf06" Oct 31 19:59:32.196401 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.196330 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-365b4309-n4dmb" Oct 31 19:59:32.480951 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.480839 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45"] Oct 31 19:59:32.481541 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.481524 4910 cpu_manager.go:414] "containerMap: removing container" podUID="4bf64223-cc5f-4a12-99b2-af2b1248bdbe" containerName="revoke-keys" Oct 31 19:59:32.481600 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.481541 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bf64223-cc5f-4a12-99b2-af2b1248bdbe" containerName="revoke-keys" Oct 31 19:59:32.481673 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.481664 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bf64223-cc5f-4a12-99b2-af2b1248bdbe" containerName="revoke-keys" Oct 31 19:59:32.485534 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.485512 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.493848 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.493818 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-ae4ce2-openshift-default-dockercfg-btj6n\"" Oct 31 19:59:32.533890 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.533852 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45"] Oct 31 19:59:32.682141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.674762 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da37695-41cb-4698-9658-c5f7eb993220-istio-podinfo\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.682141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.680853 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da37695-41cb-4698-9658-c5f7eb993220-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.682141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.680903 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fg69\" (UniqueName: \"kubernetes.io/projected/0da37695-41cb-4698-9658-c5f7eb993220-kube-api-access-5fg69\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.682141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.681056 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-istio-data\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.682141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.681105 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da37695-41cb-4698-9658-c5f7eb993220-istio-token\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.682141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.681162 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-workload-certs\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.682141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.681246 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-istio-envoy\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.682141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.681285 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-workload-socket\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.682141 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.681347 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-credential-socket\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.782510 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.782426 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da37695-41cb-4698-9658-c5f7eb993220-istio-token\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.782510 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.782491 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-workload-certs\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.782776 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.782546 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-istio-envoy\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.782776 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.782576 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-workload-socket\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.782776 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.782638 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-credential-socket\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.782776 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.782773 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da37695-41cb-4698-9658-c5f7eb993220-istio-podinfo\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.783035 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.782979 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da37695-41cb-4698-9658-c5f7eb993220-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.783100 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.783046 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-workload-socket\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.783193 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.783167 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-credential-socket\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.783333 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.783191 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fg69\" (UniqueName: \"kubernetes.io/projected/0da37695-41cb-4698-9658-c5f7eb993220-kube-api-access-5fg69\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.783416 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.783332 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-workload-certs\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.783416 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.783352 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-istio-data\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.783664 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.783642 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da37695-41cb-4698-9658-c5f7eb993220-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.783794 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.783774 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-istio-data\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.785707 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.785679 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-istio-envoy\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.785850 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.785832 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da37695-41cb-4698-9658-c5f7eb993220-istio-podinfo\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.796718 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.796693 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da37695-41cb-4698-9658-c5f7eb993220-istio-token\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.796807 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.796761 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fg69\" (UniqueName: \"kubernetes.io/projected/0da37695-41cb-4698-9658-c5f7eb993220-kube-api-access-5fg69\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:32.801613 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:32.801592 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:33.129963 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:59:33.129885 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da37695_41cb_4698_9658_c5f7eb993220.slice/crio-6ca579c81973f4eb2ce857bfbb3a91ddf06b9cd478e1353f4a667d6bdf37702d WatchSource:0}: Error finding container 6ca579c81973f4eb2ce857bfbb3a91ddf06b9cd478e1353f4a667d6bdf37702d: Status 404 returned error can't find the container with id 6ca579c81973f4eb2ce857bfbb3a91ddf06b9cd478e1353f4a667d6bdf37702d Oct 31 19:59:33.133360 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:33.133322 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:59:33.133472 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:33.133410 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:59:33.133472 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:33.133454 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:59:33.133686 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:33.133660 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45"] Oct 31 19:59:33.208052 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:33.208022 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" event={"ID":"0da37695-41cb-4698-9658-c5f7eb993220","Type":"ContainerStarted","Data":"6ca579c81973f4eb2ce857bfbb3a91ddf06b9cd478e1353f4a667d6bdf37702d"} Oct 31 19:59:33.766255 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:33.766225 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x"] Oct 31 19:59:33.773207 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:33.773121 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x"] Oct 31 19:59:33.773374 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:33.773248 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x" Oct 31 19:59:33.779759 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:33.779729 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 19:59:33.785151 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:33.785124 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 19:59:33.800865 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:33.800835 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4e58a4c-f1c1-422c-8b1c-dd3e09da4713-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x\" (UID: \"a4e58a4c-f1c1-422c-8b1c-dd3e09da4713\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x" Oct 31 19:59:33.801085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:33.801067 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9w47\" (UniqueName: \"kubernetes.io/projected/a4e58a4c-f1c1-422c-8b1c-dd3e09da4713-kube-api-access-w9w47\") pod \"maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x\" (UID: \"a4e58a4c-f1c1-422c-8b1c-dd3e09da4713\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x" Oct 31 19:59:33.801162 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:33.801093 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e58a4c-f1c1-422c-8b1c-dd3e09da4713-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x\" (UID: \"a4e58a4c-f1c1-422c-8b1c-dd3e09da4713\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x" Oct 31 19:59:33.903041 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:33.902993 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9w47\" (UniqueName: \"kubernetes.io/projected/a4e58a4c-f1c1-422c-8b1c-dd3e09da4713-kube-api-access-w9w47\") pod \"maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x\" (UID: \"a4e58a4c-f1c1-422c-8b1c-dd3e09da4713\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x" Oct 31 19:59:33.903297 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:33.903274 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e58a4c-f1c1-422c-8b1c-dd3e09da4713-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x\" (UID: \"a4e58a4c-f1c1-422c-8b1c-dd3e09da4713\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x" Oct 31 19:59:33.903682 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:33.903658 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4e58a4c-f1c1-422c-8b1c-dd3e09da4713-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x\" (UID: \"a4e58a4c-f1c1-422c-8b1c-dd3e09da4713\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x" Oct 31 19:59:33.904457 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:33.904433 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4e58a4c-f1c1-422c-8b1c-dd3e09da4713-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x\" (UID: \"a4e58a4c-f1c1-422c-8b1c-dd3e09da4713\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x" Oct 31 19:59:33.906513 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:33.906487 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e58a4c-f1c1-422c-8b1c-dd3e09da4713-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x\" (UID: \"a4e58a4c-f1c1-422c-8b1c-dd3e09da4713\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x" Oct 31 19:59:33.916093 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:33.916071 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9w47\" (UniqueName: \"kubernetes.io/projected/a4e58a4c-f1c1-422c-8b1c-dd3e09da4713-kube-api-access-w9w47\") pod \"maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x\" (UID: \"a4e58a4c-f1c1-422c-8b1c-dd3e09da4713\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x" Oct 31 19:59:34.114103 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:34.114071 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x" Oct 31 19:59:34.330943 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:34.320808 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" event={"ID":"0da37695-41cb-4698-9658-c5f7eb993220","Type":"ContainerStarted","Data":"712c27d054fe6dd01c9c3772672019bec42c0ef4a1c320afbb09566c7166d5d7"} Oct 31 19:59:34.369994 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:34.369035 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" podStartSLOduration=2.369011944 podStartE2EDuration="2.369011944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:34.345856974 +0000 UTC m=+2408.708677365" watchObservedRunningTime="2026-10-31 19:59:34.369011944 +0000 UTC m=+2408.731832317" Oct 31 19:59:34.414600 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:34.414566 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x"] Oct 31 19:59:34.416682 ip-10-0-142-116 kubenswrapper[4910]: W1031 19:59:34.416659 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e58a4c_f1c1_422c_8b1c_dd3e09da4713.slice/crio-a6e9377e720ec696ff2a879567075c26eac2eef73e2a18c9aa621c87e86345d2 WatchSource:0}: Error finding container a6e9377e720ec696ff2a879567075c26eac2eef73e2a18c9aa621c87e86345d2: Status 404 returned error can't find the container with id a6e9377e720ec696ff2a879567075c26eac2eef73e2a18c9aa621c87e86345d2 Oct 31 19:59:34.802822 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:34.802489 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:34.804085 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:34.804056 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 31 19:59:34.804230 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:34.804106 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" podUID="0da37695-41cb-4698-9658-c5f7eb993220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 31 19:59:35.353937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:35.353394 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x" event={"ID":"a4e58a4c-f1c1-422c-8b1c-dd3e09da4713","Type":"ContainerStarted","Data":"b63abc0f1d534035528a66bfdb1d88c4b4cc8f326d5fbfb4dcd57d4fc0764b00"} Oct 31 19:59:35.353937 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:35.353456 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x" event={"ID":"a4e58a4c-f1c1-422c-8b1c-dd3e09da4713","Type":"ContainerStarted","Data":"a6e9377e720ec696ff2a879567075c26eac2eef73e2a18c9aa621c87e86345d2"} Oct 31 19:59:35.374585 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:35.374297 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x" podStartSLOduration=2.374278568 podStartE2EDuration="2.374278568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:35.373681027 +0000 UTC m=+2409.736501404" watchObservedRunningTime="2026-10-31 19:59:35.374278568 +0000 UTC m=+2409.737098942" Oct 31 19:59:35.802401 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:35.802367 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 31 19:59:35.802585 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:35.802416 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" podUID="0da37695-41cb-4698-9658-c5f7eb993220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 31 19:59:36.365449 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:36.365264 4910 generic.go:356] "Generic (PLEG): container finished" podID="a4e58a4c-f1c1-422c-8b1c-dd3e09da4713" containerID="b63abc0f1d534035528a66bfdb1d88c4b4cc8f326d5fbfb4dcd57d4fc0764b00" exitCode=0 Oct 31 19:59:36.366624 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:36.366419 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x" event={"ID":"a4e58a4c-f1c1-422c-8b1c-dd3e09da4713","Type":"ContainerDied","Data":"b63abc0f1d534035528a66bfdb1d88c4b4cc8f326d5fbfb4dcd57d4fc0764b00"} Oct 31 19:59:36.802020 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:36.801945 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 31 19:59:36.802226 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:36.802203 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" podUID="0da37695-41cb-4698-9658-c5f7eb993220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 31 19:59:37.403253 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.403214 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh"] Oct 31 19:59:37.410208 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.409947 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh"] Oct 31 19:59:37.410208 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.410062 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.496119 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.495375 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-istio-envoy\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.496119 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.495439 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-workload-certs\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.496119 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.495572 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-workload-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.496119 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.495683 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpf2l\" (UniqueName: \"kubernetes.io/projected/a27f1f16-7533-4079-9702-f1ed916a760f-kube-api-access-mpf2l\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.496119 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.495728 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a27f1f16-7533-4079-9702-f1ed916a760f-istio-token\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.496119 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.495814 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a27f1f16-7533-4079-9702-f1ed916a760f-istio-podinfo\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.496119 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.495936 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-credential-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.496119 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.496117 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a27f1f16-7533-4079-9702-f1ed916a760f-istiod-ca-cert\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.496698 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.496217 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-istio-data\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.597455 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.597409 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-credential-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.597651 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.597578 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a27f1f16-7533-4079-9702-f1ed916a760f-istiod-ca-cert\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.597725 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.597658 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-istio-data\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.597780 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.597742 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-istio-envoy\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.597843 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.597777 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-workload-certs\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.597843 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.597821 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-workload-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.597981 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.597887 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpf2l\" (UniqueName: \"kubernetes.io/projected/a27f1f16-7533-4079-9702-f1ed916a760f-kube-api-access-mpf2l\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.597981 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.597936 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a27f1f16-7533-4079-9702-f1ed916a760f-istio-token\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.598091 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.597986 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a27f1f16-7533-4079-9702-f1ed916a760f-istio-podinfo\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.599595 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.598717 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-credential-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.599595 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.599014 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-workload-certs\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.599595 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.599107 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a27f1f16-7533-4079-9702-f1ed916a760f-istiod-ca-cert\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.599595 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.599557 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-istio-data\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.601237 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.601210 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-workload-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.601407 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.601389 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-istio-envoy\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.602153 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.602128 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a27f1f16-7533-4079-9702-f1ed916a760f-istio-podinfo\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.615609 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.614524 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpf2l\" (UniqueName: \"kubernetes.io/projected/a27f1f16-7533-4079-9702-f1ed916a760f-kube-api-access-mpf2l\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.641459 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.615976 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a27f1f16-7533-4079-9702-f1ed916a760f-istio-token\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.767500 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.767242 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:37.802308 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.802279 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 31 19:59:37.802443 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.802329 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" podUID="0da37695-41cb-4698-9658-c5f7eb993220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 31 19:59:37.847938 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.847121 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x" Oct 31 19:59:37.903126 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.902566 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4e58a4c-f1c1-422c-8b1c-dd3e09da4713-kube-api-access-w9w47\" (UniqueName: \"kubernetes.io/projected/a4e58a4c-f1c1-422c-8b1c-dd3e09da4713-kube-api-access-w9w47\") pod \"a4e58a4c-f1c1-422c-8b1c-dd3e09da4713\" (UID: \"a4e58a4c-f1c1-422c-8b1c-dd3e09da4713\") " Oct 31 19:59:37.903126 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.902647 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4e58a4c-f1c1-422c-8b1c-dd3e09da4713-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e58a4c-f1c1-422c-8b1c-dd3e09da4713-maas-api-xks-ca\") pod \"a4e58a4c-f1c1-422c-8b1c-dd3e09da4713\" (UID: \"a4e58a4c-f1c1-422c-8b1c-dd3e09da4713\") " Oct 31 19:59:37.903126 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.902731 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4e58a4c-f1c1-422c-8b1c-dd3e09da4713-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4e58a4c-f1c1-422c-8b1c-dd3e09da4713-maas-api-openshift-ca\") pod \"a4e58a4c-f1c1-422c-8b1c-dd3e09da4713\" (UID: \"a4e58a4c-f1c1-422c-8b1c-dd3e09da4713\") " Oct 31 19:59:37.904576 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.904538 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4e58a4c-f1c1-422c-8b1c-dd3e09da4713-maas-api-openshift-ca" pod "a4e58a4c-f1c1-422c-8b1c-dd3e09da4713" (UID: "a4e58a4c-f1c1-422c-8b1c-dd3e09da4713"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 19:59:37.906890 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.906845 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e58a4c-f1c1-422c-8b1c-dd3e09da4713-kube-api-access-w9w47" pod "a4e58a4c-f1c1-422c-8b1c-dd3e09da4713" (UID: "a4e58a4c-f1c1-422c-8b1c-dd3e09da4713"). InnerVolumeSpecName "kube-api-access-w9w47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 19:59:37.909449 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:37.909396 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4e58a4c-f1c1-422c-8b1c-dd3e09da4713-maas-api-xks-ca" pod "a4e58a4c-f1c1-422c-8b1c-dd3e09da4713" (UID: "a4e58a4c-f1c1-422c-8b1c-dd3e09da4713"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 19:59:38.008058 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:38.007968 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9w47\" (UniqueName: \"kubernetes.io/projected/a4e58a4c-f1c1-422c-8b1c-dd3e09da4713-kube-api-access-w9w47\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:38.008058 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:38.008013 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4e58a4c-f1c1-422c-8b1c-dd3e09da4713-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:38.008058 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:38.008029 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4e58a4c-f1c1-422c-8b1c-dd3e09da4713-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 19:59:38.304476 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:38.303569 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:59:38.304476 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:38.303676 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:59:38.304476 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:38.303719 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:59:38.396487 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:38.369960 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh"] Oct 31 19:59:38.849231 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:38.849189 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 31 19:59:38.849851 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:38.849811 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" podUID="0da37695-41cb-4698-9658-c5f7eb993220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 31 19:59:38.975145 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:38.975029 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" event={"ID":"a27f1f16-7533-4079-9702-f1ed916a760f","Type":"ContainerStarted","Data":"5454af44879cb4ff131d50103dbe876f9e2acbca2ed1c5402921c0372e98f9b3"} Oct 31 19:59:38.981339 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:38.979387 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x" event={"ID":"a4e58a4c-f1c1-422c-8b1c-dd3e09da4713","Type":"ContainerDied","Data":"a6e9377e720ec696ff2a879567075c26eac2eef73e2a18c9aa621c87e86345d2"} Oct 31 19:59:38.981339 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:38.979451 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6e9377e720ec696ff2a879567075c26eac2eef73e2a18c9aa621c87e86345d2" Oct 31 19:59:38.981339 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:38.979566 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x" Oct 31 19:59:39.857168 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:39.850360 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 31 19:59:39.857168 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:39.850435 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" podUID="0da37695-41cb-4698-9658-c5f7eb993220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 31 19:59:40.022985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:40.021289 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" event={"ID":"a27f1f16-7533-4079-9702-f1ed916a760f","Type":"ContainerStarted","Data":"3da1a71f7335760b52d31877b6a085968cdd8ff10a4c56790ab6cc92d60e73f8"} Oct 31 19:59:40.127988 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:40.127079 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" podStartSLOduration=3.127055934 podStartE2EDuration="3.127055934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:40.066612068 +0000 UTC m=+2414.429432441" watchObservedRunningTime="2026-10-31 19:59:40.127055934 +0000 UTC m=+2414.489876308" Oct 31 19:59:40.774940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:40.769283 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:40.774940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:40.770817 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 19:59:40.774940 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:40.770878 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" podUID="a27f1f16-7533-4079-9702-f1ed916a760f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 19:59:40.802287 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:40.802244 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 31 19:59:40.802480 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:40.802322 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" podUID="0da37695-41cb-4698-9658-c5f7eb993220" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 31 19:59:41.043249 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:41.042228 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": read tcp 10.132.0.2:58560->10.132.0.240:15021: read: connection reset by peer" start-of-body= Oct 31 19:59:41.043249 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:41.042284 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" podUID="67aa01d5-e453-4154-88f3-ef247ff5949e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": read tcp 10.132.0.2:58560->10.132.0.240:15021: read: connection reset by peer" Oct 31 19:59:41.774936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:41.771290 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 19:59:41.774936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:41.771351 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" podUID="a27f1f16-7533-4079-9702-f1ed916a760f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 19:59:41.808694 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:41.808654 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:41.809272 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:41.809244 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:41.810317 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:41.810294 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 19:59:42.071839 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.071534 4910 generic.go:356] "Generic (PLEG): container finished" podID="67aa01d5-e453-4154-88f3-ef247ff5949e" containerID="c232ac7fd865f3a8f4a69550980f2ebc3d30d8d83d6476d6e434c13949825ac9" exitCode=0 Oct 31 19:59:42.071839 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.071610 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" event={"ID":"67aa01d5-e453-4154-88f3-ef247ff5949e","Type":"ContainerDied","Data":"c232ac7fd865f3a8f4a69550980f2ebc3d30d8d83d6476d6e434c13949825ac9"} Oct 31 19:59:42.072448 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.072246 4910 scope.go:122] "RemoveContainer" containerID="c232ac7fd865f3a8f4a69550980f2ebc3d30d8d83d6476d6e434c13949825ac9" Oct 31 19:59:42.074835 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.074797 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:59:42.074974 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.074903 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:59:42.075017 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.074980 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:59:42.503986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.501689 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf"] Oct 31 19:59:42.503986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.503560 4910 cpu_manager.go:414] "containerMap: removing container" podUID="a4e58a4c-f1c1-422c-8b1c-dd3e09da4713" containerName="revoke-keys" Oct 31 19:59:42.503986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.503584 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4e58a4c-f1c1-422c-8b1c-dd3e09da4713" containerName="revoke-keys" Oct 31 19:59:42.503986 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.503839 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4e58a4c-f1c1-422c-8b1c-dd3e09da4713" containerName="revoke-keys" Oct 31 19:59:42.517293 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.508512 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.644852 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.573261 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf"] Oct 31 19:59:42.679934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.678241 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-workload-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.679934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.678390 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-workload-certs\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.679934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.678445 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-envoy\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.679934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.678501 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-podinfo\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.679934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.678557 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a521083-7066-42d2-8997-7e97d4a7ef3e-istiod-ca-cert\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.679934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.678630 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-credential-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.679934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.678691 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-token\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.679934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.678764 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-data\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.679934 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.678814 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x4wr\" (UniqueName: \"kubernetes.io/projected/4a521083-7066-42d2-8997-7e97d4a7ef3e-kube-api-access-4x4wr\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.773984 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.773355 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 19:59:42.773984 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.773420 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" podUID="a27f1f16-7533-4079-9702-f1ed916a760f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 19:59:42.776612 ip-10-0-142-116 kubenswrapper[4910]: E1031 19:59:42.776532 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 19:59:42.781973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.780865 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-workload-certs\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.781973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.780948 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-envoy\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.781973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.781044 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-podinfo\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.781973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.781109 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a521083-7066-42d2-8997-7e97d4a7ef3e-istiod-ca-cert\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.781973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.781188 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-credential-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.781973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.781243 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-token\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.781973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.781301 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-workload-certs\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.781973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.781319 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-data\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.781973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.781362 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x4wr\" (UniqueName: \"kubernetes.io/projected/4a521083-7066-42d2-8997-7e97d4a7ef3e-kube-api-access-4x4wr\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.781973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.781504 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-workload-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.781973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.781750 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a521083-7066-42d2-8997-7e97d4a7ef3e-istiod-ca-cert\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.781973 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.781931 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-workload-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.782738 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.782208 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-data\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.782738 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.782543 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-credential-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.783824 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.783797 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-envoy\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.791067 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.791037 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-podinfo\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.796079 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.796050 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-token\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.796683 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.796660 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x4wr\" (UniqueName: \"kubernetes.io/projected/4a521083-7066-42d2-8997-7e97d4a7ef3e-kube-api-access-4x4wr\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:42.871971 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:42.870317 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:43.216025 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:43.215988 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" event={"ID":"67aa01d5-e453-4154-88f3-ef247ff5949e","Type":"ContainerStarted","Data":"c17d01aff8aa897d15f86c63aec420d3290295427ebfc647848f37c4333f6cfe"} Oct 31 19:59:43.798117 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:43.796465 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 19:59:43.798117 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:43.796526 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" podUID="a27f1f16-7533-4079-9702-f1ed916a760f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 19:59:43.840148 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:43.810538 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:59:43.840148 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:43.810640 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:59:43.840148 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:43.810690 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 19:59:44.002067 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:44.001721 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 31 19:59:44.002067 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:44.001783 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" podUID="67aa01d5-e453-4154-88f3-ef247ff5949e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 31 19:59:44.190600 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:44.190512 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf"] Oct 31 19:59:44.190898 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:44.190881 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:59:44.316407 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:44.316326 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" event={"ID":"4a521083-7066-42d2-8997-7e97d4a7ef3e","Type":"ContainerStarted","Data":"5d012232d45e12a4ad65ae1d680234f057f3a5e896687e249cdef5869bb9fa9a"} Oct 31 19:59:44.483995 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:44.480980 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:44.493034 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:44.492988 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:44.538048 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:44.538008 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 19:59:44.556128 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:44.556071 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 19:59:44.691939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:44.690000 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" podUID="8fa2f1d6-abaa-4c73-8fdc-b91691bfb030" containerName="main" probeResult="failure" output="Get \"https://10.132.0.243:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:44.774988 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:44.774319 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 19:59:44.774988 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:44.774385 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" podUID="a27f1f16-7533-4079-9702-f1ed916a760f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 19:59:45.000099 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:44.999362 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 31 19:59:45.000099 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:44.999438 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" podUID="67aa01d5-e453-4154-88f3-ef247ff5949e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 31 19:59:45.478939 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:45.477946 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" event={"ID":"4a521083-7066-42d2-8997-7e97d4a7ef3e","Type":"ContainerStarted","Data":"998030f55119599ed804de17e403d16183f6e93439c046fd91588ed5a7148824"} Oct 31 19:59:45.531988 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:45.519520 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" podStartSLOduration=3.51950076 podStartE2EDuration="3.51950076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:45.511571295 +0000 UTC m=+2419.874391668" watchObservedRunningTime="2026-10-31 19:59:45.51950076 +0000 UTC m=+2419.882321133" Oct 31 19:59:45.802601 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:45.802507 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 19:59:45.802853 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:45.802821 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" podUID="a27f1f16-7533-4079-9702-f1ed916a760f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 19:59:45.912648 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:45.882503 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:45.912851 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:45.912775 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 19:59:45.912944 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:45.912838 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" podUID="4a521083-7066-42d2-8997-7e97d4a7ef3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 19:59:45.996936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:45.989432 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 31 19:59:45.996936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:45.989495 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" podUID="67aa01d5-e453-4154-88f3-ef247ff5949e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 31 19:59:46.806052 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:46.806014 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 19:59:46.806637 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:46.806602 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" podUID="a27f1f16-7533-4079-9702-f1ed916a760f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 19:59:46.882933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:46.882021 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 19:59:46.882933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:46.882092 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" podUID="4a521083-7066-42d2-8997-7e97d4a7ef3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 19:59:47.005936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:47.004301 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 31 19:59:47.005936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:47.004367 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" podUID="67aa01d5-e453-4154-88f3-ef247ff5949e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 31 19:59:47.212543 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:47.212029 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 19:59:47.769933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:47.769299 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Oct 31 19:59:47.769933 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:47.769359 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" podUID="a27f1f16-7533-4079-9702-f1ed916a760f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Oct 31 19:59:47.873352 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:47.873312 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 19:59:47.874014 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:47.873970 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" podUID="4a521083-7066-42d2-8997-7e97d4a7ef3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 19:59:47.989566 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:47.989529 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 31 19:59:47.989774 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:47.989582 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" podUID="67aa01d5-e453-4154-88f3-ef247ff5949e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 31 19:59:48.787342 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:48.787308 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:48.787794 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:48.787769 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:48.788895 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:48.788869 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 19:59:48.872936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:48.871646 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 19:59:48.872936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:48.871708 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" podUID="4a521083-7066-42d2-8997-7e97d4a7ef3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 19:59:48.999936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:48.996346 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 31 19:59:48.999936 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:48.996408 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" podUID="67aa01d5-e453-4154-88f3-ef247ff5949e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 31 19:59:49.871484 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:49.871448 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 19:59:49.871691 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:49.871512 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" podUID="4a521083-7066-42d2-8997-7e97d4a7ef3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 19:59:50.193985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:50.190090 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:59:50.193985 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:50.191880 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:59:50.207897 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:50.205184 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 19:59:50.871117 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:50.871077 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 19:59:50.871328 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:50.871159 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" podUID="4a521083-7066-42d2-8997-7e97d4a7ef3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 19:59:51.870772 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:51.870727 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Oct 31 19:59:51.871277 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:51.870796 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" podUID="4a521083-7066-42d2-8997-7e97d4a7ef3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Oct 31 19:59:52.876754 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:52.876721 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:52.877385 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:52.877124 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 19:59:52.877895 ip-10-0-142-116 kubenswrapper[4910]: I1031 19:59:52.877869 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 20:00:00.319008 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.318967 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9"] Oct 31 20:00:00.335192 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.335163 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9" Oct 31 20:00:00.380884 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.380809 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 20:00:00.468402 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.468350 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5vl4\" (UniqueName: \"kubernetes.io/projected/1b31bd96-7e0b-448c-b45f-825fe60427ca-kube-api-access-f5vl4\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9\" (UID: \"1b31bd96-7e0b-448c-b45f-825fe60427ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9" Oct 31 20:00:00.468702 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.468680 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b31bd96-7e0b-448c-b45f-825fe60427ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9\" (UID: \"1b31bd96-7e0b-448c-b45f-825fe60427ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9" Oct 31 20:00:00.484987 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.484959 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5"] Oct 31 20:00:00.500024 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.499997 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5" Oct 31 20:00:00.514960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.509590 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56"] Oct 31 20:00:00.524250 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.523665 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56" Oct 31 20:00:00.531260 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.531233 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk"] Oct 31 20:00:00.545567 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.543280 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk" Oct 31 20:00:00.561162 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.561136 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9"] Oct 31 20:00:00.571020 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.570405 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5vl4\" (UniqueName: \"kubernetes.io/projected/1b31bd96-7e0b-448c-b45f-825fe60427ca-kube-api-access-f5vl4\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9\" (UID: \"1b31bd96-7e0b-448c-b45f-825fe60427ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9" Oct 31 20:00:00.571020 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.570471 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b31bd96-7e0b-448c-b45f-825fe60427ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9\" (UID: \"1b31bd96-7e0b-448c-b45f-825fe60427ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9" Oct 31 20:00:00.577795 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.577766 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b31bd96-7e0b-448c-b45f-825fe60427ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9\" (UID: \"1b31bd96-7e0b-448c-b45f-825fe60427ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9" Oct 31 20:00:00.597553 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.590059 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5vl4\" (UniqueName: \"kubernetes.io/projected/1b31bd96-7e0b-448c-b45f-825fe60427ca-kube-api-access-f5vl4\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9\" (UID: \"1b31bd96-7e0b-448c-b45f-825fe60427ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9" Oct 31 20:00:00.636907 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.636866 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5"] Oct 31 20:00:00.645392 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.645356 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s"] Oct 31 20:00:00.651154 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.651126 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s" Oct 31 20:00:00.655064 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.655028 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6"] Oct 31 20:00:00.660828 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.660804 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm"] Oct 31 20:00:00.661315 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.661047 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6" Oct 31 20:00:00.671252 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.671232 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk"] Oct 31 20:00:00.671489 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.671474 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm" Oct 31 20:00:00.672077 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.672051 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpfm7\" (UniqueName: \"kubernetes.io/projected/99ef85db-e8f9-4826-be8c-6afdfdea13c0-kube-api-access-qpfm7\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk\" (UID: \"99ef85db-e8f9-4826-be8c-6afdfdea13c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk" Oct 31 20:00:00.672191 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.672094 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6fzm\" (UniqueName: \"kubernetes.io/projected/40440b1b-cbf1-4978-b545-fe368a4157c2-kube-api-access-g6fzm\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56\" (UID: \"40440b1b-cbf1-4978-b545-fe368a4157c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56" Oct 31 20:00:00.672263 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.672240 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99ef85db-e8f9-4826-be8c-6afdfdea13c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk\" (UID: \"99ef85db-e8f9-4826-be8c-6afdfdea13c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk" Oct 31 20:00:00.672356 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.672341 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75310598-a5d8-4012-993a-fecce8281289-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5\" (UID: \"75310598-a5d8-4012-993a-fecce8281289\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5" Oct 31 20:00:00.672438 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.672426 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmz7m\" (UniqueName: \"kubernetes.io/projected/75310598-a5d8-4012-993a-fecce8281289-kube-api-access-kmz7m\") pod \"maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5\" (UID: \"75310598-a5d8-4012-993a-fecce8281289\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5" Oct 31 20:00:00.672518 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.672506 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40440b1b-cbf1-4978-b545-fe368a4157c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56\" (UID: \"40440b1b-cbf1-4978-b545-fe368a4157c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56" Oct 31 20:00:00.678737 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.678717 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb"] Oct 31 20:00:00.684479 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.684455 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb" Oct 31 20:00:00.684928 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.684890 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6"] Oct 31 20:00:00.689020 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.688997 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9" Oct 31 20:00:00.689585 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.689566 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn"] Oct 31 20:00:00.698665 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.698640 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn"] Oct 31 20:00:00.698787 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.698746 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn" Oct 31 20:00:00.700025 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.700003 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb"] Oct 31 20:00:00.710620 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.710228 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c"] Oct 31 20:00:00.717241 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.717046 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c" Oct 31 20:00:00.717241 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.717187 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s"] Oct 31 20:00:00.732857 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.731714 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56"] Oct 31 20:00:00.746140 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.743891 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm"] Oct 31 20:00:00.750291 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.750081 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c"] Oct 31 20:00:00.779674 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.777962 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12bf7929-e3b2-406e-9095-450b003c84e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c\" (UID: \"12bf7929-e3b2-406e-9095-450b003c84e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c" Oct 31 20:00:00.779674 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.778023 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b76hx\" (UniqueName: \"kubernetes.io/projected/e9d2cb4a-9f51-448e-bf66-af95955df697-kube-api-access-b76hx\") pod \"maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm\" (UID: \"e9d2cb4a-9f51-448e-bf66-af95955df697\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm" Oct 31 20:00:00.779674 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.778103 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d30713d-b20b-47f1-9aee-087c061f7835-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s\" (UID: \"3d30713d-b20b-47f1-9aee-087c061f7835\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s" Oct 31 20:00:00.779674 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.778177 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqlsm\" (UniqueName: \"kubernetes.io/projected/f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33-kube-api-access-xqlsm\") pod \"maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6\" (UID: \"f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6" Oct 31 20:00:00.779674 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.778211 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpfm7\" (UniqueName: \"kubernetes.io/projected/99ef85db-e8f9-4826-be8c-6afdfdea13c0-kube-api-access-qpfm7\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk\" (UID: \"99ef85db-e8f9-4826-be8c-6afdfdea13c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk" Oct 31 20:00:00.779674 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.778247 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6fzm\" (UniqueName: \"kubernetes.io/projected/40440b1b-cbf1-4978-b545-fe368a4157c2-kube-api-access-g6fzm\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56\" (UID: \"40440b1b-cbf1-4978-b545-fe368a4157c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56" Oct 31 20:00:00.779674 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.778277 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6\" (UID: \"f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6" Oct 31 20:00:00.779674 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.778350 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh26l\" (UniqueName: \"kubernetes.io/projected/12bf7929-e3b2-406e-9095-450b003c84e9-kube-api-access-jh26l\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c\" (UID: \"12bf7929-e3b2-406e-9095-450b003c84e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c" Oct 31 20:00:00.779674 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.778418 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d2cb4a-9f51-448e-bf66-af95955df697-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm\" (UID: \"e9d2cb4a-9f51-448e-bf66-af95955df697\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm" Oct 31 20:00:00.779674 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.778445 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55ch5\" (UniqueName: \"kubernetes.io/projected/3d30713d-b20b-47f1-9aee-087c061f7835-kube-api-access-55ch5\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s\" (UID: \"3d30713d-b20b-47f1-9aee-087c061f7835\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s" Oct 31 20:00:00.779674 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.778476 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99ef85db-e8f9-4826-be8c-6afdfdea13c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk\" (UID: \"99ef85db-e8f9-4826-be8c-6afdfdea13c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk" Oct 31 20:00:00.779674 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.778506 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a336785-37c0-4fa9-a100-4872bc215b03-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891280-r2lcb\" (UID: \"5a336785-37c0-4fa9-a100-4872bc215b03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb" Oct 31 20:00:00.779674 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.778553 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hzvz\" (UniqueName: \"kubernetes.io/projected/5a336785-37c0-4fa9-a100-4872bc215b03-kube-api-access-6hzvz\") pod \"maas-api-key-cleanup-29891280-r2lcb\" (UID: \"5a336785-37c0-4fa9-a100-4872bc215b03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb" Oct 31 20:00:00.779674 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.778633 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75310598-a5d8-4012-993a-fecce8281289-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5\" (UID: \"75310598-a5d8-4012-993a-fecce8281289\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5" Oct 31 20:00:00.779674 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.778720 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmz7m\" (UniqueName: \"kubernetes.io/projected/75310598-a5d8-4012-993a-fecce8281289-kube-api-access-kmz7m\") pod \"maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5\" (UID: \"75310598-a5d8-4012-993a-fecce8281289\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5" Oct 31 20:00:00.779674 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.778783 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce6c21a-41e1-4e7e-ac6d-2cc90582803a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn\" (UID: \"cce6c21a-41e1-4e7e-ac6d-2cc90582803a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn" Oct 31 20:00:00.779674 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.778827 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40440b1b-cbf1-4978-b545-fe368a4157c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56\" (UID: \"40440b1b-cbf1-4978-b545-fe368a4157c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56" Oct 31 20:00:00.779674 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.778858 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9tl6\" (UniqueName: \"kubernetes.io/projected/cce6c21a-41e1-4e7e-ac6d-2cc90582803a-kube-api-access-w9tl6\") pod \"maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn\" (UID: \"cce6c21a-41e1-4e7e-ac6d-2cc90582803a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn" Oct 31 20:00:00.783575 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.780154 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99ef85db-e8f9-4826-be8c-6afdfdea13c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk\" (UID: \"99ef85db-e8f9-4826-be8c-6afdfdea13c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk" Oct 31 20:00:00.783575 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.780452 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75310598-a5d8-4012-993a-fecce8281289-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5\" (UID: \"75310598-a5d8-4012-993a-fecce8281289\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5" Oct 31 20:00:00.783575 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.781015 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40440b1b-cbf1-4978-b545-fe368a4157c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56\" (UID: \"40440b1b-cbf1-4978-b545-fe368a4157c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56" Oct 31 20:00:00.804967 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.804573 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmz7m\" (UniqueName: \"kubernetes.io/projected/75310598-a5d8-4012-993a-fecce8281289-kube-api-access-kmz7m\") pod \"maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5\" (UID: \"75310598-a5d8-4012-993a-fecce8281289\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5" Oct 31 20:00:00.804967 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.804707 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpfm7\" (UniqueName: \"kubernetes.io/projected/99ef85db-e8f9-4826-be8c-6afdfdea13c0-kube-api-access-qpfm7\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk\" (UID: \"99ef85db-e8f9-4826-be8c-6afdfdea13c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk" Oct 31 20:00:00.804967 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.804857 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6fzm\" (UniqueName: \"kubernetes.io/projected/40440b1b-cbf1-4978-b545-fe368a4157c2-kube-api-access-g6fzm\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56\" (UID: \"40440b1b-cbf1-4978-b545-fe368a4157c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56" Oct 31 20:00:00.876037 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.875944 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5" Oct 31 20:00:00.879820 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.879773 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12bf7929-e3b2-406e-9095-450b003c84e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c\" (UID: \"12bf7929-e3b2-406e-9095-450b003c84e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c" Oct 31 20:00:00.879891 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.879839 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b76hx\" (UniqueName: \"kubernetes.io/projected/e9d2cb4a-9f51-448e-bf66-af95955df697-kube-api-access-b76hx\") pod \"maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm\" (UID: \"e9d2cb4a-9f51-448e-bf66-af95955df697\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm" Oct 31 20:00:00.879891 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.879862 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d30713d-b20b-47f1-9aee-087c061f7835-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s\" (UID: \"3d30713d-b20b-47f1-9aee-087c061f7835\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s" Oct 31 20:00:00.880071 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.880046 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqlsm\" (UniqueName: \"kubernetes.io/projected/f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33-kube-api-access-xqlsm\") pod \"maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6\" (UID: \"f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6" Oct 31 20:00:00.880151 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.880101 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6\" (UID: \"f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6" Oct 31 20:00:00.880214 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.880178 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh26l\" (UniqueName: \"kubernetes.io/projected/12bf7929-e3b2-406e-9095-450b003c84e9-kube-api-access-jh26l\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c\" (UID: \"12bf7929-e3b2-406e-9095-450b003c84e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c" Oct 31 20:00:00.880285 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.880252 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d2cb4a-9f51-448e-bf66-af95955df697-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm\" (UID: \"e9d2cb4a-9f51-448e-bf66-af95955df697\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm" Oct 31 20:00:00.880345 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.880282 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55ch5\" (UniqueName: \"kubernetes.io/projected/3d30713d-b20b-47f1-9aee-087c061f7835-kube-api-access-55ch5\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s\" (UID: \"3d30713d-b20b-47f1-9aee-087c061f7835\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s" Oct 31 20:00:00.880345 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.880322 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a336785-37c0-4fa9-a100-4872bc215b03-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891280-r2lcb\" (UID: \"5a336785-37c0-4fa9-a100-4872bc215b03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb" Oct 31 20:00:00.880466 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.880369 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hzvz\" (UniqueName: \"kubernetes.io/projected/5a336785-37c0-4fa9-a100-4872bc215b03-kube-api-access-6hzvz\") pod \"maas-api-key-cleanup-29891280-r2lcb\" (UID: \"5a336785-37c0-4fa9-a100-4872bc215b03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb" Oct 31 20:00:00.880574 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.880551 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce6c21a-41e1-4e7e-ac6d-2cc90582803a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn\" (UID: \"cce6c21a-41e1-4e7e-ac6d-2cc90582803a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn" Oct 31 20:00:00.880652 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.880609 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9tl6\" (UniqueName: \"kubernetes.io/projected/cce6c21a-41e1-4e7e-ac6d-2cc90582803a-kube-api-access-w9tl6\") pod \"maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn\" (UID: \"cce6c21a-41e1-4e7e-ac6d-2cc90582803a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn" Oct 31 20:00:00.880728 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.880653 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12bf7929-e3b2-406e-9095-450b003c84e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c\" (UID: \"12bf7929-e3b2-406e-9095-450b003c84e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c" Oct 31 20:00:00.880728 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.880691 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d30713d-b20b-47f1-9aee-087c061f7835-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s\" (UID: \"3d30713d-b20b-47f1-9aee-087c061f7835\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s" Oct 31 20:00:00.881065 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.881045 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d2cb4a-9f51-448e-bf66-af95955df697-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm\" (UID: \"e9d2cb4a-9f51-448e-bf66-af95955df697\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm" Oct 31 20:00:00.881419 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.881392 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce6c21a-41e1-4e7e-ac6d-2cc90582803a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn\" (UID: \"cce6c21a-41e1-4e7e-ac6d-2cc90582803a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn" Oct 31 20:00:00.881499 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.881462 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6\" (UID: \"f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6" Oct 31 20:00:00.882028 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.881986 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a336785-37c0-4fa9-a100-4872bc215b03-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891280-r2lcb\" (UID: \"5a336785-37c0-4fa9-a100-4872bc215b03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb" Oct 31 20:00:00.892506 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.892485 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqlsm\" (UniqueName: \"kubernetes.io/projected/f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33-kube-api-access-xqlsm\") pod \"maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6\" (UID: \"f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6" Oct 31 20:00:00.892609 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.892554 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b76hx\" (UniqueName: \"kubernetes.io/projected/e9d2cb4a-9f51-448e-bf66-af95955df697-kube-api-access-b76hx\") pod \"maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm\" (UID: \"e9d2cb4a-9f51-448e-bf66-af95955df697\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm" Oct 31 20:00:00.895135 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.895111 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9tl6\" (UniqueName: \"kubernetes.io/projected/cce6c21a-41e1-4e7e-ac6d-2cc90582803a-kube-api-access-w9tl6\") pod \"maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn\" (UID: \"cce6c21a-41e1-4e7e-ac6d-2cc90582803a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn" Oct 31 20:00:00.896374 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.896351 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56" Oct 31 20:00:00.896374 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.896357 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh26l\" (UniqueName: \"kubernetes.io/projected/12bf7929-e3b2-406e-9095-450b003c84e9-kube-api-access-jh26l\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c\" (UID: \"12bf7929-e3b2-406e-9095-450b003c84e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c" Oct 31 20:00:00.897823 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.897801 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55ch5\" (UniqueName: \"kubernetes.io/projected/3d30713d-b20b-47f1-9aee-087c061f7835-kube-api-access-55ch5\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s\" (UID: \"3d30713d-b20b-47f1-9aee-087c061f7835\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s" Oct 31 20:00:00.897952 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.897902 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hzvz\" (UniqueName: \"kubernetes.io/projected/5a336785-37c0-4fa9-a100-4872bc215b03-kube-api-access-6hzvz\") pod \"maas-api-key-cleanup-29891280-r2lcb\" (UID: \"5a336785-37c0-4fa9-a100-4872bc215b03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb" Oct 31 20:00:00.939345 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.939308 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk" Oct 31 20:00:00.982167 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.982121 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s" Oct 31 20:00:00.991698 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:00.991337 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6" Oct 31 20:00:01.001538 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:01.001510 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm" Oct 31 20:00:01.020217 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:01.019121 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9"] Oct 31 20:00:01.032279 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:00:01.031674 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b31bd96_7e0b_448c_b45f_825fe60427ca.slice/crio-0a970d3f58b290fcb251cdecf8ca2069a80856bf170db92bf79157495172eeda WatchSource:0}: Error finding container 0a970d3f58b290fcb251cdecf8ca2069a80856bf170db92bf79157495172eeda: Status 404 returned error can't find the container with id 0a970d3f58b290fcb251cdecf8ca2069a80856bf170db92bf79157495172eeda Oct 31 20:00:01.066840 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:01.066654 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb" Oct 31 20:00:01.096675 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:01.087482 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn" Oct 31 20:00:01.105486 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:01.105440 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c" Oct 31 20:00:01.190770 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:01.190694 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9" event={"ID":"1b31bd96-7e0b-448c-b45f-825fe60427ca","Type":"ContainerStarted","Data":"0a970d3f58b290fcb251cdecf8ca2069a80856bf170db92bf79157495172eeda"} Oct 31 20:00:01.217153 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:00:01.215804 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75310598_a5d8_4012_993a_fecce8281289.slice/crio-db4da3efca8054104ae0f4227651ec5d0a758e392951f0a352e038ebbe0c012b WatchSource:0}: Error finding container db4da3efca8054104ae0f4227651ec5d0a758e392951f0a352e038ebbe0c012b: Status 404 returned error can't find the container with id db4da3efca8054104ae0f4227651ec5d0a758e392951f0a352e038ebbe0c012b Oct 31 20:00:01.217153 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:01.216001 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5"] Oct 31 20:00:01.307404 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:00:01.307370 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ef85db_e8f9_4826_be8c_6afdfdea13c0.slice/crio-b4bac5b7ec0c5e6cbc6e1288b5ccc189a9e24446f9f9fc33b747642b9fcbb968 WatchSource:0}: Error finding container b4bac5b7ec0c5e6cbc6e1288b5ccc189a9e24446f9f9fc33b747642b9fcbb968: Status 404 returned error can't find the container with id b4bac5b7ec0c5e6cbc6e1288b5ccc189a9e24446f9f9fc33b747642b9fcbb968 Oct 31 20:00:01.310601 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:01.310425 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk"] Oct 31 20:00:01.464136 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:00:01.464104 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40440b1b_cbf1_4978_b545_fe368a4157c2.slice/crio-4a284eb1eee4e76b6ed73bc8447d04f2c30da320ac810db19295864daf5ad1c3 WatchSource:0}: Error finding container 4a284eb1eee4e76b6ed73bc8447d04f2c30da320ac810db19295864daf5ad1c3: Status 404 returned error can't find the container with id 4a284eb1eee4e76b6ed73bc8447d04f2c30da320ac810db19295864daf5ad1c3 Oct 31 20:00:01.470047 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:01.468141 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56"] Oct 31 20:00:01.639255 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:00:01.625181 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d30713d_b20b_47f1_9aee_087c061f7835.slice/crio-9ca4eacfcb59120200509bf7a74c80b7838723dd3d464fedfa1e3a739e8ac94d WatchSource:0}: Error finding container 9ca4eacfcb59120200509bf7a74c80b7838723dd3d464fedfa1e3a739e8ac94d: Status 404 returned error can't find the container with id 9ca4eacfcb59120200509bf7a74c80b7838723dd3d464fedfa1e3a739e8ac94d Oct 31 20:00:01.663416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:01.652648 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s"] Oct 31 20:00:01.674376 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:01.674344 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm"] Oct 31 20:00:01.827737 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:01.827701 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6"] Oct 31 20:00:01.921986 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:01.921946 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn"] Oct 31 20:00:01.937006 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:00:01.936789 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce6c21a_41e1_4e7e_ac6d_2cc90582803a.slice/crio-0eac30d61eb3c5fc5af9e9021b357bce9485c17352e47c46a28f9a3cc5375def WatchSource:0}: Error finding container 0eac30d61eb3c5fc5af9e9021b357bce9485c17352e47c46a28f9a3cc5375def: Status 404 returned error can't find the container with id 0eac30d61eb3c5fc5af9e9021b357bce9485c17352e47c46a28f9a3cc5375def Oct 31 20:00:01.945681 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:01.945513 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb"] Oct 31 20:00:01.972209 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:01.972184 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c"] Oct 31 20:00:02.202174 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:02.202094 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c" event={"ID":"12bf7929-e3b2-406e-9095-450b003c84e9","Type":"ContainerStarted","Data":"07d79bffdae8bbb19185b5941601b7dd1502999ff0cbc227c607cae62a3a769b"} Oct 31 20:00:02.203576 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:02.203539 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6" event={"ID":"f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33","Type":"ContainerStarted","Data":"3aefaef8ec7297e554aa1480f18faaada77c0cdada99a8316500f3340c5afb9d"} Oct 31 20:00:02.205588 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:02.205554 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56" event={"ID":"40440b1b-cbf1-4978-b545-fe368a4157c2","Type":"ContainerStarted","Data":"7f8822dfbc589f991bf3e791bb94c3c140bfb8ee83502132fbd3dac325f0fb0c"} Oct 31 20:00:02.205694 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:02.205596 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56" event={"ID":"40440b1b-cbf1-4978-b545-fe368a4157c2","Type":"ContainerStarted","Data":"4a284eb1eee4e76b6ed73bc8447d04f2c30da320ac810db19295864daf5ad1c3"} Oct 31 20:00:02.208452 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:02.208430 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5" event={"ID":"75310598-a5d8-4012-993a-fecce8281289","Type":"ContainerStarted","Data":"264caa6df4d5fe8ac7023921cef198c3ff5b83d6d23a984d41e375e3a6cd940c"} Oct 31 20:00:02.208545 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:02.208477 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5" event={"ID":"75310598-a5d8-4012-993a-fecce8281289","Type":"ContainerStarted","Data":"db4da3efca8054104ae0f4227651ec5d0a758e392951f0a352e038ebbe0c012b"} Oct 31 20:00:02.210708 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:02.210681 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk" event={"ID":"99ef85db-e8f9-4826-be8c-6afdfdea13c0","Type":"ContainerStarted","Data":"52fd22bf8bdf78dde87cdc7a3900c99ef8fe15bac649ac570f656761c6165775"} Oct 31 20:00:02.210884 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:02.210730 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk" event={"ID":"99ef85db-e8f9-4826-be8c-6afdfdea13c0","Type":"ContainerStarted","Data":"b4bac5b7ec0c5e6cbc6e1288b5ccc189a9e24446f9f9fc33b747642b9fcbb968"} Oct 31 20:00:02.212154 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:02.212127 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm" event={"ID":"e9d2cb4a-9f51-448e-bf66-af95955df697","Type":"ContainerStarted","Data":"1d446fd84496f5cac145e92af9208425c63aee6ad392cdb954c5af7aa0d31237"} Oct 31 20:00:02.213810 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:02.213788 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9" event={"ID":"1b31bd96-7e0b-448c-b45f-825fe60427ca","Type":"ContainerStarted","Data":"1a05403c299e0df60755fb4926dc55c66fbfcccf6556c8b51d53f0e1df73154c"} Oct 31 20:00:02.221759 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:02.221695 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn" event={"ID":"cce6c21a-41e1-4e7e-ac6d-2cc90582803a","Type":"ContainerStarted","Data":"0eac30d61eb3c5fc5af9e9021b357bce9485c17352e47c46a28f9a3cc5375def"} Oct 31 20:00:02.224263 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:02.224179 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s" event={"ID":"3d30713d-b20b-47f1-9aee-087c061f7835","Type":"ContainerStarted","Data":"9ca4eacfcb59120200509bf7a74c80b7838723dd3d464fedfa1e3a739e8ac94d"} Oct 31 20:00:02.228235 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:02.228168 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb" event={"ID":"5a336785-37c0-4fa9-a100-4872bc215b03","Type":"ContainerStarted","Data":"87a31f0ae427f987d3016d5cebc3ae8710e5896b6dcedc6ef24ac221226b19d1"} Oct 31 20:00:02.249975 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:02.248471 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56" podStartSLOduration=2.248454979 podStartE2EDuration="2.248454979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:02.242225829 +0000 UTC m=+2436.605046198" watchObservedRunningTime="2026-10-31 20:00:02.248454979 +0000 UTC m=+2436.611275352" Oct 31 20:00:02.274457 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:02.274409 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5" podStartSLOduration=2.27439527 podStartE2EDuration="2.27439527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:02.271429265 +0000 UTC m=+2436.634249640" watchObservedRunningTime="2026-10-31 20:00:02.27439527 +0000 UTC m=+2436.637215642" Oct 31 20:00:02.310299 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:02.310250 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9" podStartSLOduration=2.310237068 podStartE2EDuration="2.310237068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:02.306523727 +0000 UTC m=+2436.669344099" watchObservedRunningTime="2026-10-31 20:00:02.310237068 +0000 UTC m=+2436.673057440" Oct 31 20:00:02.348649 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:02.348588 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk" podStartSLOduration=2.348573581 podStartE2EDuration="2.348573581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:02.34163581 +0000 UTC m=+2436.704456181" watchObservedRunningTime="2026-10-31 20:00:02.348573581 +0000 UTC m=+2436.711393976" Oct 31 20:00:03.241873 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.241838 4910 generic.go:356] "Generic (PLEG): container finished" podID="99ef85db-e8f9-4826-be8c-6afdfdea13c0" containerID="52fd22bf8bdf78dde87cdc7a3900c99ef8fe15bac649ac570f656761c6165775" exitCode=60 Oct 31 20:00:03.242348 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.241968 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk" event={"ID":"99ef85db-e8f9-4826-be8c-6afdfdea13c0","Type":"ContainerDied","Data":"52fd22bf8bdf78dde87cdc7a3900c99ef8fe15bac649ac570f656761c6165775"} Oct 31 20:00:03.242348 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.242297 4910 scope.go:122] "RemoveContainer" containerID="52fd22bf8bdf78dde87cdc7a3900c99ef8fe15bac649ac570f656761c6165775" Oct 31 20:00:03.244542 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.244412 4910 generic.go:356] "Generic (PLEG): container finished" podID="e9d2cb4a-9f51-448e-bf66-af95955df697" containerID="c69aad754a513249adc5bee5fca77ab81c6a9511709ac01751261a1643950854" exitCode=60 Oct 31 20:00:03.244542 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.244440 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm" event={"ID":"e9d2cb4a-9f51-448e-bf66-af95955df697","Type":"ContainerDied","Data":"c69aad754a513249adc5bee5fca77ab81c6a9511709ac01751261a1643950854"} Oct 31 20:00:03.244808 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.244791 4910 scope.go:122] "RemoveContainer" containerID="c69aad754a513249adc5bee5fca77ab81c6a9511709ac01751261a1643950854" Oct 31 20:00:03.246958 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.246499 4910 generic.go:356] "Generic (PLEG): container finished" podID="1b31bd96-7e0b-448c-b45f-825fe60427ca" containerID="1a05403c299e0df60755fb4926dc55c66fbfcccf6556c8b51d53f0e1df73154c" exitCode=60 Oct 31 20:00:03.246958 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.246573 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9" event={"ID":"1b31bd96-7e0b-448c-b45f-825fe60427ca","Type":"ContainerDied","Data":"1a05403c299e0df60755fb4926dc55c66fbfcccf6556c8b51d53f0e1df73154c"} Oct 31 20:00:03.246958 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.246855 4910 scope.go:122] "RemoveContainer" containerID="1a05403c299e0df60755fb4926dc55c66fbfcccf6556c8b51d53f0e1df73154c" Oct 31 20:00:03.253699 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.253676 4910 generic.go:356] "Generic (PLEG): container finished" podID="cce6c21a-41e1-4e7e-ac6d-2cc90582803a" containerID="1a373b4e270b021e6b08a7ee9d98c9071a38c851a5e6442c72a846683e41d26e" exitCode=60 Oct 31 20:00:03.253802 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.253766 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn" event={"ID":"cce6c21a-41e1-4e7e-ac6d-2cc90582803a","Type":"ContainerDied","Data":"1a373b4e270b021e6b08a7ee9d98c9071a38c851a5e6442c72a846683e41d26e"} Oct 31 20:00:03.254049 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.254026 4910 scope.go:122] "RemoveContainer" containerID="1a373b4e270b021e6b08a7ee9d98c9071a38c851a5e6442c72a846683e41d26e" Oct 31 20:00:03.263665 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.262972 4910 generic.go:356] "Generic (PLEG): container finished" podID="3d30713d-b20b-47f1-9aee-087c061f7835" containerID="dad4105cd0a4ce68b94b69e46943a7cde65ed33ef560ab344e46742158f43e67" exitCode=60 Oct 31 20:00:03.263665 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.263003 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s" event={"ID":"3d30713d-b20b-47f1-9aee-087c061f7835","Type":"ContainerDied","Data":"dad4105cd0a4ce68b94b69e46943a7cde65ed33ef560ab344e46742158f43e67"} Oct 31 20:00:03.263665 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.263296 4910 scope.go:122] "RemoveContainer" containerID="dad4105cd0a4ce68b94b69e46943a7cde65ed33ef560ab344e46742158f43e67" Oct 31 20:00:03.270717 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.269687 4910 generic.go:356] "Generic (PLEG): container finished" podID="5a336785-37c0-4fa9-a100-4872bc215b03" containerID="0203eeb42f771e4dab6f03559350ac2ca16cd26928bc5e47fb17dec8fac50bc2" exitCode=60 Oct 31 20:00:03.270717 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.269856 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb" event={"ID":"5a336785-37c0-4fa9-a100-4872bc215b03","Type":"ContainerDied","Data":"0203eeb42f771e4dab6f03559350ac2ca16cd26928bc5e47fb17dec8fac50bc2"} Oct 31 20:00:03.270717 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.270137 4910 scope.go:122] "RemoveContainer" containerID="0203eeb42f771e4dab6f03559350ac2ca16cd26928bc5e47fb17dec8fac50bc2" Oct 31 20:00:03.273806 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.273781 4910 generic.go:356] "Generic (PLEG): container finished" podID="12bf7929-e3b2-406e-9095-450b003c84e9" containerID="1c80300a8984f331ae0ecfc046c028b346d3feeb4b844dec315833dcc32d0983" exitCode=60 Oct 31 20:00:03.273999 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.273889 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c" event={"ID":"12bf7929-e3b2-406e-9095-450b003c84e9","Type":"ContainerDied","Data":"1c80300a8984f331ae0ecfc046c028b346d3feeb4b844dec315833dcc32d0983"} Oct 31 20:00:03.274144 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.274126 4910 scope.go:122] "RemoveContainer" containerID="1c80300a8984f331ae0ecfc046c028b346d3feeb4b844dec315833dcc32d0983" Oct 31 20:00:03.276138 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.276080 4910 generic.go:356] "Generic (PLEG): container finished" podID="f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33" containerID="d97a613633941c300778dec3fddbf3be2bced83a9f6a7fdb6cae116c3d07f77d" exitCode=60 Oct 31 20:00:03.276250 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.276158 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6" event={"ID":"f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33","Type":"ContainerDied","Data":"d97a613633941c300778dec3fddbf3be2bced83a9f6a7fdb6cae116c3d07f77d"} Oct 31 20:00:03.279082 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.279056 4910 scope.go:122] "RemoveContainer" containerID="d97a613633941c300778dec3fddbf3be2bced83a9f6a7fdb6cae116c3d07f77d" Oct 31 20:00:03.280515 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.280490 4910 generic.go:356] "Generic (PLEG): container finished" podID="40440b1b-cbf1-4978-b545-fe368a4157c2" containerID="7f8822dfbc589f991bf3e791bb94c3c140bfb8ee83502132fbd3dac325f0fb0c" exitCode=60 Oct 31 20:00:03.280623 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.280568 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56" event={"ID":"40440b1b-cbf1-4978-b545-fe368a4157c2","Type":"ContainerDied","Data":"7f8822dfbc589f991bf3e791bb94c3c140bfb8ee83502132fbd3dac325f0fb0c"} Oct 31 20:00:03.280798 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.280784 4910 scope.go:122] "RemoveContainer" containerID="7f8822dfbc589f991bf3e791bb94c3c140bfb8ee83502132fbd3dac325f0fb0c" Oct 31 20:00:03.282708 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.282622 4910 generic.go:356] "Generic (PLEG): container finished" podID="75310598-a5d8-4012-993a-fecce8281289" containerID="264caa6df4d5fe8ac7023921cef198c3ff5b83d6d23a984d41e375e3a6cd940c" exitCode=60 Oct 31 20:00:03.282811 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.282724 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5" event={"ID":"75310598-a5d8-4012-993a-fecce8281289","Type":"ContainerDied","Data":"264caa6df4d5fe8ac7023921cef198c3ff5b83d6d23a984d41e375e3a6cd940c"} Oct 31 20:00:03.282956 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:03.282896 4910 scope.go:122] "RemoveContainer" containerID="264caa6df4d5fe8ac7023921cef198c3ff5b83d6d23a984d41e375e3a6cd940c" Oct 31 20:00:04.293589 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:04.293507 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9" event={"ID":"1b31bd96-7e0b-448c-b45f-825fe60427ca","Type":"ContainerStarted","Data":"95d1db6eded59c9ba5da0f7b66054a19cdb474e695294e8443e4b3662809362a"} Oct 31 20:00:04.295757 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:04.295729 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn" event={"ID":"cce6c21a-41e1-4e7e-ac6d-2cc90582803a","Type":"ContainerStarted","Data":"03fcf416b47b16c4baa4d16cc2beaa463cf92264e666cb64bae0be0355ce1d86"} Oct 31 20:00:04.297612 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:04.297591 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s" event={"ID":"3d30713d-b20b-47f1-9aee-087c061f7835","Type":"ContainerStarted","Data":"8424c8e4a72c4f86febb235858e09dd4ff1a422d445db6f9092f52d16b8d98c8"} Oct 31 20:00:04.299697 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:04.299669 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb" event={"ID":"5a336785-37c0-4fa9-a100-4872bc215b03","Type":"ContainerStarted","Data":"0bbf8ca4958847b51aedc2cf8dcfd37de48f3b57d412b9b348db2e3c6c046aa1"} Oct 31 20:00:04.301864 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:04.301841 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c" event={"ID":"12bf7929-e3b2-406e-9095-450b003c84e9","Type":"ContainerStarted","Data":"0bfef8656a6612389c532f83e655329c0987b566d7f462277c304acb0d50e1ec"} Oct 31 20:00:04.303872 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:04.303847 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6" event={"ID":"f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33","Type":"ContainerStarted","Data":"fa35982c77cee87c2b28b39861f7e7ed492f8361da9f452603af156d0428a86f"} Oct 31 20:00:04.305684 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:04.305664 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56" event={"ID":"40440b1b-cbf1-4978-b545-fe368a4157c2","Type":"ContainerStarted","Data":"ee9429d64c32e0616d5fca9aa9d4ddc7e6e2950c3c17443ff17b89c91275f3a1"} Oct 31 20:00:04.307518 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:04.307496 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5" event={"ID":"75310598-a5d8-4012-993a-fecce8281289","Type":"ContainerStarted","Data":"c72b8791a4a67a4787f72cf1d34323d792da1eeae81f9502eb9cb485a30dc432"} Oct 31 20:00:04.309523 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:04.309501 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk" event={"ID":"99ef85db-e8f9-4826-be8c-6afdfdea13c0","Type":"ContainerStarted","Data":"1598d24b3baf87b8ade7038af4d19ee637b70a545abfb64eac668a79ed0cdb7b"} Oct 31 20:00:04.311250 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:04.311230 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm" event={"ID":"e9d2cb4a-9f51-448e-bf66-af95955df697","Type":"ContainerStarted","Data":"23baa35298e5b6767ffbfd4bb50a7ec510aea2a68e36966ca2e6d7ff27ed69e0"} Oct 31 20:00:04.376446 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:04.376363 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c" podStartSLOduration=4.376348608 podStartE2EDuration="4.376348608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:04.372555205 +0000 UTC m=+2438.735375578" watchObservedRunningTime="2026-10-31 20:00:04.376348608 +0000 UTC m=+2438.739169025" Oct 31 20:00:04.395628 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:04.395573 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb" podStartSLOduration=4.395560842 podStartE2EDuration="4.395560842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:04.393651047 +0000 UTC m=+2438.756471418" watchObservedRunningTime="2026-10-31 20:00:04.395560842 +0000 UTC m=+2438.758381211" Oct 31 20:00:04.489528 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:04.489469 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s" podStartSLOduration=4.489451853 podStartE2EDuration="4.489451853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:04.458386059 +0000 UTC m=+2438.821206431" watchObservedRunningTime="2026-10-31 20:00:04.489451853 +0000 UTC m=+2438.852272226" Oct 31 20:00:04.523227 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:04.523023 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6" podStartSLOduration=4.523003039 podStartE2EDuration="4.523003039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:04.506575637 +0000 UTC m=+2438.869396011" watchObservedRunningTime="2026-10-31 20:00:04.523003039 +0000 UTC m=+2438.885823413" Oct 31 20:00:04.571122 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:04.569594 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm" podStartSLOduration=4.569574697 podStartE2EDuration="4.569574697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:04.537637189 +0000 UTC m=+2438.900457562" watchObservedRunningTime="2026-10-31 20:00:04.569574697 +0000 UTC m=+2438.932395073" Oct 31 20:00:04.619441 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:04.619382 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn" podStartSLOduration=4.6193631360000005 podStartE2EDuration="4.619363136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:04.598374546 +0000 UTC m=+2438.961194919" watchObservedRunningTime="2026-10-31 20:00:04.619363136 +0000 UTC m=+2438.982183509" Oct 31 20:00:05.324330 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.324290 4910 generic.go:356] "Generic (PLEG): container finished" podID="3d30713d-b20b-47f1-9aee-087c061f7835" containerID="8424c8e4a72c4f86febb235858e09dd4ff1a422d445db6f9092f52d16b8d98c8" exitCode=60 Oct 31 20:00:05.324794 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.324376 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s" event={"ID":"3d30713d-b20b-47f1-9aee-087c061f7835","Type":"ContainerDied","Data":"8424c8e4a72c4f86febb235858e09dd4ff1a422d445db6f9092f52d16b8d98c8"} Oct 31 20:00:05.324794 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.324446 4910 scope.go:122] "RemoveContainer" containerID="dad4105cd0a4ce68b94b69e46943a7cde65ed33ef560ab344e46742158f43e67" Oct 31 20:00:05.324939 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.324876 4910 scope.go:122] "RemoveContainer" containerID="8424c8e4a72c4f86febb235858e09dd4ff1a422d445db6f9092f52d16b8d98c8" Oct 31 20:00:05.325354 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:05.325327 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s_odh-ai-gateway-infra(3d30713d-b20b-47f1-9aee-087c061f7835)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s" podUID="3d30713d-b20b-47f1-9aee-087c061f7835" Oct 31 20:00:05.327878 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.327853 4910 generic.go:356] "Generic (PLEG): container finished" podID="5a336785-37c0-4fa9-a100-4872bc215b03" containerID="0bbf8ca4958847b51aedc2cf8dcfd37de48f3b57d412b9b348db2e3c6c046aa1" exitCode=60 Oct 31 20:00:05.328066 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.328039 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb" event={"ID":"5a336785-37c0-4fa9-a100-4872bc215b03","Type":"ContainerDied","Data":"0bbf8ca4958847b51aedc2cf8dcfd37de48f3b57d412b9b348db2e3c6c046aa1"} Oct 31 20:00:05.328391 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.328261 4910 scope.go:122] "RemoveContainer" containerID="0bbf8ca4958847b51aedc2cf8dcfd37de48f3b57d412b9b348db2e3c6c046aa1" Oct 31 20:00:05.328671 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:05.328646 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891280-r2lcb_odh-ai-gateway-infra(5a336785-37c0-4fa9-a100-4872bc215b03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb" podUID="5a336785-37c0-4fa9-a100-4872bc215b03" Oct 31 20:00:05.331704 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.331675 4910 generic.go:356] "Generic (PLEG): container finished" podID="12bf7929-e3b2-406e-9095-450b003c84e9" containerID="0bfef8656a6612389c532f83e655329c0987b566d7f462277c304acb0d50e1ec" exitCode=60 Oct 31 20:00:05.331829 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.331752 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c" event={"ID":"12bf7929-e3b2-406e-9095-450b003c84e9","Type":"ContainerDied","Data":"0bfef8656a6612389c532f83e655329c0987b566d7f462277c304acb0d50e1ec"} Oct 31 20:00:05.332026 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.332011 4910 scope.go:122] "RemoveContainer" containerID="0bfef8656a6612389c532f83e655329c0987b566d7f462277c304acb0d50e1ec" Oct 31 20:00:05.332320 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:05.332298 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c_odh-ai-gateway-infra(12bf7929-e3b2-406e-9095-450b003c84e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c" podUID="12bf7929-e3b2-406e-9095-450b003c84e9" Oct 31 20:00:05.334114 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.334085 4910 generic.go:356] "Generic (PLEG): container finished" podID="f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33" containerID="fa35982c77cee87c2b28b39861f7e7ed492f8361da9f452603af156d0428a86f" exitCode=60 Oct 31 20:00:05.334213 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.334179 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6" event={"ID":"f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33","Type":"ContainerDied","Data":"fa35982c77cee87c2b28b39861f7e7ed492f8361da9f452603af156d0428a86f"} Oct 31 20:00:05.334585 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.334566 4910 scope.go:122] "RemoveContainer" containerID="fa35982c77cee87c2b28b39861f7e7ed492f8361da9f452603af156d0428a86f" Oct 31 20:00:05.334935 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:05.334893 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6_odh-ai-gateway-infra(f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6" podUID="f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33" Oct 31 20:00:05.336452 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.336429 4910 generic.go:356] "Generic (PLEG): container finished" podID="40440b1b-cbf1-4978-b545-fe368a4157c2" containerID="ee9429d64c32e0616d5fca9aa9d4ddc7e6e2950c3c17443ff17b89c91275f3a1" exitCode=60 Oct 31 20:00:05.336551 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.336515 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56" event={"ID":"40440b1b-cbf1-4978-b545-fe368a4157c2","Type":"ContainerDied","Data":"ee9429d64c32e0616d5fca9aa9d4ddc7e6e2950c3c17443ff17b89c91275f3a1"} Oct 31 20:00:05.336923 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.336706 4910 scope.go:122] "RemoveContainer" containerID="ee9429d64c32e0616d5fca9aa9d4ddc7e6e2950c3c17443ff17b89c91275f3a1" Oct 31 20:00:05.337062 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:05.337038 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56_odh-ai-gateway-infra(40440b1b-cbf1-4978-b545-fe368a4157c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56" podUID="40440b1b-cbf1-4978-b545-fe368a4157c2" Oct 31 20:00:05.345292 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.345274 4910 scope.go:122] "RemoveContainer" containerID="0203eeb42f771e4dab6f03559350ac2ca16cd26928bc5e47fb17dec8fac50bc2" Oct 31 20:00:05.346201 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.346061 4910 generic.go:356] "Generic (PLEG): container finished" podID="75310598-a5d8-4012-993a-fecce8281289" containerID="c72b8791a4a67a4787f72cf1d34323d792da1eeae81f9502eb9cb485a30dc432" exitCode=60 Oct 31 20:00:05.346421 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.346209 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5" event={"ID":"75310598-a5d8-4012-993a-fecce8281289","Type":"ContainerDied","Data":"c72b8791a4a67a4787f72cf1d34323d792da1eeae81f9502eb9cb485a30dc432"} Oct 31 20:00:05.346580 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.346560 4910 scope.go:122] "RemoveContainer" containerID="c72b8791a4a67a4787f72cf1d34323d792da1eeae81f9502eb9cb485a30dc432" Oct 31 20:00:05.347223 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:05.347197 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5_odh-ai-gateway-infra(75310598-a5d8-4012-993a-fecce8281289)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5" podUID="75310598-a5d8-4012-993a-fecce8281289" Oct 31 20:00:05.350132 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.350110 4910 generic.go:356] "Generic (PLEG): container finished" podID="99ef85db-e8f9-4826-be8c-6afdfdea13c0" containerID="1598d24b3baf87b8ade7038af4d19ee637b70a545abfb64eac668a79ed0cdb7b" exitCode=60 Oct 31 20:00:05.350241 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.350188 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk" event={"ID":"99ef85db-e8f9-4826-be8c-6afdfdea13c0","Type":"ContainerDied","Data":"1598d24b3baf87b8ade7038af4d19ee637b70a545abfb64eac668a79ed0cdb7b"} Oct 31 20:00:05.350417 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.350398 4910 scope.go:122] "RemoveContainer" containerID="1598d24b3baf87b8ade7038af4d19ee637b70a545abfb64eac668a79ed0cdb7b" Oct 31 20:00:05.350736 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:05.350711 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk_odh-ai-gateway-infra(99ef85db-e8f9-4826-be8c-6afdfdea13c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk" podUID="99ef85db-e8f9-4826-be8c-6afdfdea13c0" Oct 31 20:00:05.352703 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.352616 4910 generic.go:356] "Generic (PLEG): container finished" podID="e9d2cb4a-9f51-448e-bf66-af95955df697" containerID="23baa35298e5b6767ffbfd4bb50a7ec510aea2a68e36966ca2e6d7ff27ed69e0" exitCode=60 Oct 31 20:00:05.352949 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.352671 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm" event={"ID":"e9d2cb4a-9f51-448e-bf66-af95955df697","Type":"ContainerDied","Data":"23baa35298e5b6767ffbfd4bb50a7ec510aea2a68e36966ca2e6d7ff27ed69e0"} Oct 31 20:00:05.355640 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.355233 4910 generic.go:356] "Generic (PLEG): container finished" podID="1b31bd96-7e0b-448c-b45f-825fe60427ca" containerID="95d1db6eded59c9ba5da0f7b66054a19cdb474e695294e8443e4b3662809362a" exitCode=60 Oct 31 20:00:05.355640 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.355320 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9" event={"ID":"1b31bd96-7e0b-448c-b45f-825fe60427ca","Type":"ContainerDied","Data":"95d1db6eded59c9ba5da0f7b66054a19cdb474e695294e8443e4b3662809362a"} Oct 31 20:00:05.356694 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.356599 4910 scope.go:122] "RemoveContainer" containerID="23baa35298e5b6767ffbfd4bb50a7ec510aea2a68e36966ca2e6d7ff27ed69e0" Oct 31 20:00:05.356867 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.356844 4910 scope.go:122] "RemoveContainer" containerID="95d1db6eded59c9ba5da0f7b66054a19cdb474e695294e8443e4b3662809362a" Oct 31 20:00:05.356985 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:05.356961 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm_odh-ai-gateway-infra(e9d2cb4a-9f51-448e-bf66-af95955df697)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm" podUID="e9d2cb4a-9f51-448e-bf66-af95955df697" Oct 31 20:00:05.357208 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:05.357185 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9_odh-ai-gateway-infra(1b31bd96-7e0b-448c-b45f-825fe60427ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9" podUID="1b31bd96-7e0b-448c-b45f-825fe60427ca" Oct 31 20:00:05.358434 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.358413 4910 generic.go:356] "Generic (PLEG): container finished" podID="cce6c21a-41e1-4e7e-ac6d-2cc90582803a" containerID="03fcf416b47b16c4baa4d16cc2beaa463cf92264e666cb64bae0be0355ce1d86" exitCode=60 Oct 31 20:00:05.358611 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.358442 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn" event={"ID":"cce6c21a-41e1-4e7e-ac6d-2cc90582803a","Type":"ContainerDied","Data":"03fcf416b47b16c4baa4d16cc2beaa463cf92264e666cb64bae0be0355ce1d86"} Oct 31 20:00:05.358688 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.358673 4910 scope.go:122] "RemoveContainer" containerID="03fcf416b47b16c4baa4d16cc2beaa463cf92264e666cb64bae0be0355ce1d86" Oct 31 20:00:05.358924 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:05.358893 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn_odh-ai-gateway-infra(cce6c21a-41e1-4e7e-ac6d-2cc90582803a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn" podUID="cce6c21a-41e1-4e7e-ac6d-2cc90582803a" Oct 31 20:00:05.364711 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.364693 4910 scope.go:122] "RemoveContainer" containerID="1c80300a8984f331ae0ecfc046c028b346d3feeb4b844dec315833dcc32d0983" Oct 31 20:00:05.379894 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.379870 4910 scope.go:122] "RemoveContainer" containerID="d97a613633941c300778dec3fddbf3be2bced83a9f6a7fdb6cae116c3d07f77d" Oct 31 20:00:05.394031 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.394007 4910 scope.go:122] "RemoveContainer" containerID="7f8822dfbc589f991bf3e791bb94c3c140bfb8ee83502132fbd3dac325f0fb0c" Oct 31 20:00:05.408706 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.408686 4910 scope.go:122] "RemoveContainer" containerID="264caa6df4d5fe8ac7023921cef198c3ff5b83d6d23a984d41e375e3a6cd940c" Oct 31 20:00:05.424090 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.424065 4910 scope.go:122] "RemoveContainer" containerID="52fd22bf8bdf78dde87cdc7a3900c99ef8fe15bac649ac570f656761c6165775" Oct 31 20:00:05.442396 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.442372 4910 scope.go:122] "RemoveContainer" containerID="c69aad754a513249adc5bee5fca77ab81c6a9511709ac01751261a1643950854" Oct 31 20:00:05.460474 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.460454 4910 scope.go:122] "RemoveContainer" containerID="1a05403c299e0df60755fb4926dc55c66fbfcccf6556c8b51d53f0e1df73154c" Oct 31 20:00:05.475928 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:05.475896 4910 scope.go:122] "RemoveContainer" containerID="1a373b4e270b021e6b08a7ee9d98c9071a38c851a5e6442c72a846683e41d26e" Oct 31 20:00:06.371613 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:06.371585 4910 scope.go:122] "RemoveContainer" containerID="95d1db6eded59c9ba5da0f7b66054a19cdb474e695294e8443e4b3662809362a" Oct 31 20:00:06.372085 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:06.372013 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9_odh-ai-gateway-infra(1b31bd96-7e0b-448c-b45f-825fe60427ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9" podUID="1b31bd96-7e0b-448c-b45f-825fe60427ca" Oct 31 20:00:06.373408 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:06.373382 4910 scope.go:122] "RemoveContainer" containerID="03fcf416b47b16c4baa4d16cc2beaa463cf92264e666cb64bae0be0355ce1d86" Oct 31 20:00:06.373803 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:06.373780 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn_odh-ai-gateway-infra(cce6c21a-41e1-4e7e-ac6d-2cc90582803a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn" podUID="cce6c21a-41e1-4e7e-ac6d-2cc90582803a" Oct 31 20:00:06.375201 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:06.375182 4910 scope.go:122] "RemoveContainer" containerID="8424c8e4a72c4f86febb235858e09dd4ff1a422d445db6f9092f52d16b8d98c8" Oct 31 20:00:06.375513 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:06.375492 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s_odh-ai-gateway-infra(3d30713d-b20b-47f1-9aee-087c061f7835)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s" podUID="3d30713d-b20b-47f1-9aee-087c061f7835" Oct 31 20:00:06.377279 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:06.377257 4910 scope.go:122] "RemoveContainer" containerID="0bbf8ca4958847b51aedc2cf8dcfd37de48f3b57d412b9b348db2e3c6c046aa1" Oct 31 20:00:06.377639 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:06.377616 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891280-r2lcb_odh-ai-gateway-infra(5a336785-37c0-4fa9-a100-4872bc215b03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb" podUID="5a336785-37c0-4fa9-a100-4872bc215b03" Oct 31 20:00:06.379120 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:06.379098 4910 scope.go:122] "RemoveContainer" containerID="0bfef8656a6612389c532f83e655329c0987b566d7f462277c304acb0d50e1ec" Oct 31 20:00:06.379458 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:06.379435 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c_odh-ai-gateway-infra(12bf7929-e3b2-406e-9095-450b003c84e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c" podUID="12bf7929-e3b2-406e-9095-450b003c84e9" Oct 31 20:00:06.380820 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:06.380800 4910 scope.go:122] "RemoveContainer" containerID="fa35982c77cee87c2b28b39861f7e7ed492f8361da9f452603af156d0428a86f" Oct 31 20:00:06.381148 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:06.381128 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6_odh-ai-gateway-infra(f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6" podUID="f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33" Oct 31 20:00:06.382455 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:06.382439 4910 scope.go:122] "RemoveContainer" containerID="ee9429d64c32e0616d5fca9aa9d4ddc7e6e2950c3c17443ff17b89c91275f3a1" Oct 31 20:00:06.382761 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:06.382730 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56_odh-ai-gateway-infra(40440b1b-cbf1-4978-b545-fe368a4157c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56" podUID="40440b1b-cbf1-4978-b545-fe368a4157c2" Oct 31 20:00:06.384110 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:06.384087 4910 scope.go:122] "RemoveContainer" containerID="c72b8791a4a67a4787f72cf1d34323d792da1eeae81f9502eb9cb485a30dc432" Oct 31 20:00:06.384357 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:06.384336 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5_odh-ai-gateway-infra(75310598-a5d8-4012-993a-fecce8281289)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5" podUID="75310598-a5d8-4012-993a-fecce8281289" Oct 31 20:00:06.387073 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:06.387053 4910 scope.go:122] "RemoveContainer" containerID="1598d24b3baf87b8ade7038af4d19ee637b70a545abfb64eac668a79ed0cdb7b" Oct 31 20:00:06.387475 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:06.387300 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk_odh-ai-gateway-infra(99ef85db-e8f9-4826-be8c-6afdfdea13c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk" podUID="99ef85db-e8f9-4826-be8c-6afdfdea13c0" Oct 31 20:00:06.389073 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:06.389050 4910 scope.go:122] "RemoveContainer" containerID="23baa35298e5b6767ffbfd4bb50a7ec510aea2a68e36966ca2e6d7ff27ed69e0" Oct 31 20:00:06.389391 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:06.389364 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm_odh-ai-gateway-infra(e9d2cb4a-9f51-448e-bf66-af95955df697)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm" podUID="e9d2cb4a-9f51-448e-bf66-af95955df697" Oct 31 20:00:07.241963 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:07.233539 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4"] Oct 31 20:00:07.241963 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:07.233831 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4" podUID="1031afac-c58c-48e9-9ef6-b6fa69f0f29e" containerName="maas-api" containerID="cri-o://3ee8fee18463d8bd6a0bf36fd32a40bc00b6d01c806cfe472f296b98d66bd14a" gracePeriod=30 Oct 31 20:00:07.482884 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:07.482398 4910 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1031afac_c58c_48e9_9ef6_b6fa69f0f29e.slice/crio-3ee8fee18463d8bd6a0bf36fd32a40bc00b6d01c806cfe472f296b98d66bd14a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1031afac_c58c_48e9_9ef6_b6fa69f0f29e.slice/crio-conmon-3ee8fee18463d8bd6a0bf36fd32a40bc00b6d01c806cfe472f296b98d66bd14a.scope\": RecentStats: unable to find data in memory cache]" Oct 31 20:00:07.506848 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:07.506754 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4" podUID="1031afac-c58c-48e9-9ef6-b6fa69f0f29e" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.234:8443/health\": dial tcp 10.132.0.234:8443: connect: connection refused" Oct 31 20:00:08.192200 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:08.192163 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x"] Oct 31 20:00:08.192449 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:08.192430 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1f2781-vp69x"] Oct 31 20:00:08.242420 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:08.242389 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4" Oct 31 20:00:08.267417 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:08.267372 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-maas-api-tls\") pod \"1031afac-c58c-48e9-9ef6-b6fa69f0f29e\" (UID: \"1031afac-c58c-48e9-9ef6-b6fa69f0f29e\") " Oct 31 20:00:08.267680 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:08.267659 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-kube-api-access-x4w4h\" (UniqueName: \"kubernetes.io/projected/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-kube-api-access-x4w4h\") pod \"1031afac-c58c-48e9-9ef6-b6fa69f0f29e\" (UID: \"1031afac-c58c-48e9-9ef6-b6fa69f0f29e\") " Oct 31 20:00:08.267732 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:08.267707 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-metrics-certs\") pod \"1031afac-c58c-48e9-9ef6-b6fa69f0f29e\" (UID: \"1031afac-c58c-48e9-9ef6-b6fa69f0f29e\") " Oct 31 20:00:08.267871 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:08.267855 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-openshift-service-ca\") pod \"1031afac-c58c-48e9-9ef6-b6fa69f0f29e\" (UID: \"1031afac-c58c-48e9-9ef6-b6fa69f0f29e\") " Oct 31 20:00:08.273902 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:08.270018 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-openshift-service-ca" pod "1031afac-c58c-48e9-9ef6-b6fa69f0f29e" (UID: "1031afac-c58c-48e9-9ef6-b6fa69f0f29e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:08.273902 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:08.272428 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-kube-api-access-x4w4h" pod "1031afac-c58c-48e9-9ef6-b6fa69f0f29e" (UID: "1031afac-c58c-48e9-9ef6-b6fa69f0f29e"). InnerVolumeSpecName "kube-api-access-x4w4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:08.274572 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:08.274274 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-metrics-certs" pod "1031afac-c58c-48e9-9ef6-b6fa69f0f29e" (UID: "1031afac-c58c-48e9-9ef6-b6fa69f0f29e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:00:08.277581 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:08.277547 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-maas-api-tls" pod "1031afac-c58c-48e9-9ef6-b6fa69f0f29e" (UID: "1031afac-c58c-48e9-9ef6-b6fa69f0f29e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:00:08.370174 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:08.370130 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4w4h\" (UniqueName: \"kubernetes.io/projected/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-kube-api-access-x4w4h\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:08.370174 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:08.370175 4910 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-metrics-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:08.370395 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:08.370193 4910 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-openshift-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:08.370395 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:08.370206 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1031afac-c58c-48e9-9ef6-b6fa69f0f29e-maas-api-tls\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:08.417378 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:08.417342 4910 generic.go:356] "Generic (PLEG): container finished" podID="1031afac-c58c-48e9-9ef6-b6fa69f0f29e" containerID="3ee8fee18463d8bd6a0bf36fd32a40bc00b6d01c806cfe472f296b98d66bd14a" exitCode=0 Oct 31 20:00:08.417539 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:08.417392 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4" Oct 31 20:00:08.417539 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:08.417421 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4" event={"ID":"1031afac-c58c-48e9-9ef6-b6fa69f0f29e","Type":"ContainerDied","Data":"3ee8fee18463d8bd6a0bf36fd32a40bc00b6d01c806cfe472f296b98d66bd14a"} Oct 31 20:00:08.417539 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:08.417483 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4" event={"ID":"1031afac-c58c-48e9-9ef6-b6fa69f0f29e","Type":"ContainerDied","Data":"783baeb587eeeeceaa2fcaa4c655d88f73e8713714e3ad63539ad906d2b4f2e3"} Oct 31 20:00:08.417539 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:08.417513 4910 scope.go:122] "RemoveContainer" containerID="3ee8fee18463d8bd6a0bf36fd32a40bc00b6d01c806cfe472f296b98d66bd14a" Oct 31 20:00:08.432059 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:08.432034 4910 scope.go:122] "RemoveContainer" containerID="3ee8fee18463d8bd6a0bf36fd32a40bc00b6d01c806cfe472f296b98d66bd14a" Oct 31 20:00:08.432400 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:08.432379 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ee8fee18463d8bd6a0bf36fd32a40bc00b6d01c806cfe472f296b98d66bd14a\": container with ID starting with 3ee8fee18463d8bd6a0bf36fd32a40bc00b6d01c806cfe472f296b98d66bd14a not found: ID does not exist" containerID="3ee8fee18463d8bd6a0bf36fd32a40bc00b6d01c806cfe472f296b98d66bd14a" Oct 31 20:00:08.432474 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:08.432413 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee8fee18463d8bd6a0bf36fd32a40bc00b6d01c806cfe472f296b98d66bd14a"} err="failed to get container status \"3ee8fee18463d8bd6a0bf36fd32a40bc00b6d01c806cfe472f296b98d66bd14a\": rpc error: code = NotFound desc = could not find container \"3ee8fee18463d8bd6a0bf36fd32a40bc00b6d01c806cfe472f296b98d66bd14a\": container with ID starting with 3ee8fee18463d8bd6a0bf36fd32a40bc00b6d01c806cfe472f296b98d66bd14a not found: ID does not exist" Oct 31 20:00:08.455048 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:08.454990 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4"] Oct 31 20:00:08.462990 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:08.462962 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1f2781-7574bf69cf-jd9s4"] Oct 31 20:00:09.759575 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:09.759530 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1031afac-c58c-48e9-9ef6-b6fa69f0f29e" path="/var/lib/kubelet/pods/1031afac-c58c-48e9-9ef6-b6fa69f0f29e/volumes" Oct 31 20:00:09.760309 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:09.760285 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4e58a4c-f1c1-422c-8b1c-dd3e09da4713" path="/var/lib/kubelet/pods/a4e58a4c-f1c1-422c-8b1c-dd3e09da4713/volumes" Oct 31 20:00:10.748887 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:10.748846 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16883d53-tvt6g"] Oct 31 20:00:10.750155 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:10.750129 4910 cpu_manager.go:414] "containerMap: removing container" podUID="1031afac-c58c-48e9-9ef6-b6fa69f0f29e" containerName="maas-api" Oct 31 20:00:10.750155 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:10.750152 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1031afac-c58c-48e9-9ef6-b6fa69f0f29e" containerName="maas-api" Oct 31 20:00:10.750365 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:10.750317 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="1031afac-c58c-48e9-9ef6-b6fa69f0f29e" containerName="maas-api" Oct 31 20:00:10.762224 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:10.762195 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16883d53-tvt6g" Oct 31 20:00:10.763374 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:10.762810 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16883d53-tvt6g"] Oct 31 20:00:10.770729 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:10.768817 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:00:10.871820 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:10.871780 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52f21d22-e27f-45eb-ae8f-9d971b0baa9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16883d53-tvt6g\" (UID: \"52f21d22-e27f-45eb-ae8f-9d971b0baa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16883d53-tvt6g" Oct 31 20:00:10.872082 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:10.872065 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52f21d22-e27f-45eb-ae8f-9d971b0baa9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16883d53-tvt6g\" (UID: \"52f21d22-e27f-45eb-ae8f-9d971b0baa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16883d53-tvt6g" Oct 31 20:00:10.872179 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:10.872115 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8glvb\" (UniqueName: \"kubernetes.io/projected/52f21d22-e27f-45eb-ae8f-9d971b0baa9e-kube-api-access-8glvb\") pod \"maas-api-revoke-sub-16883d53-tvt6g\" (UID: \"52f21d22-e27f-45eb-ae8f-9d971b0baa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16883d53-tvt6g" Oct 31 20:00:10.973224 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:10.973182 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52f21d22-e27f-45eb-ae8f-9d971b0baa9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16883d53-tvt6g\" (UID: \"52f21d22-e27f-45eb-ae8f-9d971b0baa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16883d53-tvt6g" Oct 31 20:00:10.973437 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:10.973420 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52f21d22-e27f-45eb-ae8f-9d971b0baa9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16883d53-tvt6g\" (UID: \"52f21d22-e27f-45eb-ae8f-9d971b0baa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16883d53-tvt6g" Oct 31 20:00:10.973500 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:10.973475 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8glvb\" (UniqueName: \"kubernetes.io/projected/52f21d22-e27f-45eb-ae8f-9d971b0baa9e-kube-api-access-8glvb\") pod \"maas-api-revoke-sub-16883d53-tvt6g\" (UID: \"52f21d22-e27f-45eb-ae8f-9d971b0baa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16883d53-tvt6g" Oct 31 20:00:10.974158 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:10.974136 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52f21d22-e27f-45eb-ae8f-9d971b0baa9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16883d53-tvt6g\" (UID: \"52f21d22-e27f-45eb-ae8f-9d971b0baa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16883d53-tvt6g" Oct 31 20:00:10.975468 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:10.975446 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52f21d22-e27f-45eb-ae8f-9d971b0baa9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16883d53-tvt6g\" (UID: \"52f21d22-e27f-45eb-ae8f-9d971b0baa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16883d53-tvt6g" Oct 31 20:00:10.986450 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:10.986423 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8glvb\" (UniqueName: \"kubernetes.io/projected/52f21d22-e27f-45eb-ae8f-9d971b0baa9e-kube-api-access-8glvb\") pod \"maas-api-revoke-sub-16883d53-tvt6g\" (UID: \"52f21d22-e27f-45eb-ae8f-9d971b0baa9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16883d53-tvt6g" Oct 31 20:00:11.085977 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:11.085946 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16883d53-tvt6g" Oct 31 20:00:11.384791 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:11.384757 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16883d53-tvt6g"] Oct 31 20:00:11.384933 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:00:11.384855 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f21d22_e27f_45eb_ae8f_9d971b0baa9e.slice/crio-d6b06e5ac03d0827635f5ac9c6fc6d641d8d72b59e430f296ff60ce64c3d394d WatchSource:0}: Error finding container d6b06e5ac03d0827635f5ac9c6fc6d641d8d72b59e430f296ff60ce64c3d394d: Status 404 returned error can't find the container with id d6b06e5ac03d0827635f5ac9c6fc6d641d8d72b59e430f296ff60ce64c3d394d Oct 31 20:00:11.456091 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:11.456058 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16883d53-tvt6g" event={"ID":"52f21d22-e27f-45eb-ae8f-9d971b0baa9e","Type":"ContainerStarted","Data":"d6b06e5ac03d0827635f5ac9c6fc6d641d8d72b59e430f296ff60ce64c3d394d"} Oct 31 20:00:12.470186 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:12.470153 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16883d53-tvt6g" event={"ID":"52f21d22-e27f-45eb-ae8f-9d971b0baa9e","Type":"ContainerStarted","Data":"172f06655ec7b0a8b8b016296ac7dad8710997201c2967786dbd85926937af76"} Oct 31 20:00:12.497066 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:12.497008 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16883d53-tvt6g" podStartSLOduration=2.4969896560000002 podStartE2EDuration="2.496989656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:12.491033064 +0000 UTC m=+2446.853853438" watchObservedRunningTime="2026-10-31 20:00:12.496989656 +0000 UTC m=+2446.859810030" Oct 31 20:00:12.718485 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:12.718439 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:00:13.481036 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:13.480995 4910 generic.go:356] "Generic (PLEG): container finished" podID="52f21d22-e27f-45eb-ae8f-9d971b0baa9e" containerID="172f06655ec7b0a8b8b016296ac7dad8710997201c2967786dbd85926937af76" exitCode=0 Oct 31 20:00:13.481563 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:13.481084 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16883d53-tvt6g" event={"ID":"52f21d22-e27f-45eb-ae8f-9d971b0baa9e","Type":"ContainerDied","Data":"172f06655ec7b0a8b8b016296ac7dad8710997201c2967786dbd85926937af76"} Oct 31 20:00:14.734964 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.732856 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16883d53-tvt6g" Oct 31 20:00:14.792441 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.792370 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52f21d22-e27f-45eb-ae8f-9d971b0baa9e-kube-api-access-8glvb\" (UniqueName: \"kubernetes.io/projected/52f21d22-e27f-45eb-ae8f-9d971b0baa9e-kube-api-access-8glvb\") pod \"52f21d22-e27f-45eb-ae8f-9d971b0baa9e\" (UID: \"52f21d22-e27f-45eb-ae8f-9d971b0baa9e\") " Oct 31 20:00:14.793744 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.793715 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52f21d22-e27f-45eb-ae8f-9d971b0baa9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52f21d22-e27f-45eb-ae8f-9d971b0baa9e-maas-api-openshift-ca\") pod \"52f21d22-e27f-45eb-ae8f-9d971b0baa9e\" (UID: \"52f21d22-e27f-45eb-ae8f-9d971b0baa9e\") " Oct 31 20:00:14.794110 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.794088 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52f21d22-e27f-45eb-ae8f-9d971b0baa9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52f21d22-e27f-45eb-ae8f-9d971b0baa9e-maas-api-xks-ca\") pod \"52f21d22-e27f-45eb-ae8f-9d971b0baa9e\" (UID: \"52f21d22-e27f-45eb-ae8f-9d971b0baa9e\") " Oct 31 20:00:14.798035 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.797889 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f21d22-e27f-45eb-ae8f-9d971b0baa9e-kube-api-access-8glvb" pod "52f21d22-e27f-45eb-ae8f-9d971b0baa9e" (UID: "52f21d22-e27f-45eb-ae8f-9d971b0baa9e"). InnerVolumeSpecName "kube-api-access-8glvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:14.798263 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.798240 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52f21d22-e27f-45eb-ae8f-9d971b0baa9e-maas-api-openshift-ca" pod "52f21d22-e27f-45eb-ae8f-9d971b0baa9e" (UID: "52f21d22-e27f-45eb-ae8f-9d971b0baa9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:14.800469 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.800441 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52f21d22-e27f-45eb-ae8f-9d971b0baa9e-maas-api-xks-ca" pod "52f21d22-e27f-45eb-ae8f-9d971b0baa9e" (UID: "52f21d22-e27f-45eb-ae8f-9d971b0baa9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:00:14.824627 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.824595 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw"] Oct 31 20:00:14.825518 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.825494 4910 cpu_manager.go:414] "containerMap: removing container" podUID="52f21d22-e27f-45eb-ae8f-9d971b0baa9e" containerName="revoke-keys" Oct 31 20:00:14.825518 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.825518 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52f21d22-e27f-45eb-ae8f-9d971b0baa9e" containerName="revoke-keys" Oct 31 20:00:14.825778 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.825762 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="52f21d22-e27f-45eb-ae8f-9d971b0baa9e" containerName="revoke-keys" Oct 31 20:00:14.830663 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.830641 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:14.838238 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.837622 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c734f5-openshift-default-dockercfg-jg24l\"" Oct 31 20:00:14.863205 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.854052 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw"] Oct 31 20:00:14.896554 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.896509 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-workload-certs\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:14.896738 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.896582 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1110e52-d926-449c-809b-be04b7d1db94-istiod-ca-cert\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:14.896738 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.896677 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpg4b\" (UniqueName: \"kubernetes.io/projected/d1110e52-d926-449c-809b-be04b7d1db94-kube-api-access-qpg4b\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:14.896738 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.896707 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1110e52-d926-449c-809b-be04b7d1db94-istio-token\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:14.896905 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.896814 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-istio-envoy\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:14.896905 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.896893 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-workload-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:14.897063 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.897030 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-credential-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:14.897125 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.897114 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-istio-data\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:14.897185 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.897139 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1110e52-d926-449c-809b-be04b7d1db94-istio-podinfo\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:14.897493 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.897465 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8glvb\" (UniqueName: \"kubernetes.io/projected/52f21d22-e27f-45eb-ae8f-9d971b0baa9e-kube-api-access-8glvb\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:14.897619 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.897500 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52f21d22-e27f-45eb-ae8f-9d971b0baa9e-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:14.897694 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:14.897626 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52f21d22-e27f-45eb-ae8f-9d971b0baa9e-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:15.022682 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.022640 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-workload-certs\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:15.022975 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.022954 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1110e52-d926-449c-809b-be04b7d1db94-istiod-ca-cert\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:15.023116 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.023099 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpg4b\" (UniqueName: \"kubernetes.io/projected/d1110e52-d926-449c-809b-be04b7d1db94-kube-api-access-qpg4b\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:15.023224 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.023212 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1110e52-d926-449c-809b-be04b7d1db94-istio-token\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:15.023427 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.023410 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-istio-envoy\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:15.023592 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.023580 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-workload-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:15.030945 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.023770 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-credential-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:15.031274 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.031243 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-istio-data\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:15.031416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.031397 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1110e52-d926-449c-809b-be04b7d1db94-istio-podinfo\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:15.037550 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.035292 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-credential-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:15.037550 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.035561 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-workload-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:15.037550 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.036251 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1110e52-d926-449c-809b-be04b7d1db94-istiod-ca-cert\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:15.037550 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.036498 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-workload-certs\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:15.037550 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.037072 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-istio-data\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:15.070519 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.070484 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-istio-envoy\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:15.071499 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.071436 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1110e52-d926-449c-809b-be04b7d1db94-istio-podinfo\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:15.080700 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.080642 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpg4b\" (UniqueName: \"kubernetes.io/projected/d1110e52-d926-449c-809b-be04b7d1db94-kube-api-access-qpg4b\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:15.111523 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.111452 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1110e52-d926-449c-809b-be04b7d1db94-istio-token\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:15.185696 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.184230 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:15.645531 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.645487 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16883d53-tvt6g" event={"ID":"52f21d22-e27f-45eb-ae8f-9d971b0baa9e","Type":"ContainerDied","Data":"d6b06e5ac03d0827635f5ac9c6fc6d641d8d72b59e430f296ff60ce64c3d394d"} Oct 31 20:00:15.645689 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.645560 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6b06e5ac03d0827635f5ac9c6fc6d641d8d72b59e430f296ff60ce64c3d394d" Oct 31 20:00:15.645689 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.645635 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16883d53-tvt6g" Oct 31 20:00:15.764318 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.763706 4910 scope.go:122] "RemoveContainer" containerID="fa35982c77cee87c2b28b39861f7e7ed492f8361da9f452603af156d0428a86f" Oct 31 20:00:15.764318 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.764018 4910 scope.go:122] "RemoveContainer" containerID="23baa35298e5b6767ffbfd4bb50a7ec510aea2a68e36966ca2e6d7ff27ed69e0" Oct 31 20:00:15.764874 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.764340 4910 scope.go:122] "RemoveContainer" containerID="8424c8e4a72c4f86febb235858e09dd4ff1a422d445db6f9092f52d16b8d98c8" Oct 31 20:00:15.848463 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.848432 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw"] Oct 31 20:00:15.849064 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.849019 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:00:15.849169 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.849104 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:00:15.849169 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:15.849152 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:00:16.680982 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:16.678210 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm" event={"ID":"e9d2cb4a-9f51-448e-bf66-af95955df697","Type":"ContainerStarted","Data":"f7b7efc4895e30026a1b8ea3e463073538f4c449221fec5d1aa025d1fb938680"} Oct 31 20:00:16.681425 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:16.681390 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s" event={"ID":"3d30713d-b20b-47f1-9aee-087c061f7835","Type":"ContainerStarted","Data":"d6f3072dc441887d53753f5964e8f80dc364bbd4bfa35d9ffaf40c6e8c7f913e"} Oct 31 20:00:16.683692 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:16.683655 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6" event={"ID":"f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33","Type":"ContainerStarted","Data":"3474beb7340ad76cd5f0e27fab6040b282ba828ac8bdca4182f100b53f193c28"} Oct 31 20:00:16.685622 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:16.685590 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" event={"ID":"d1110e52-d926-449c-809b-be04b7d1db94","Type":"ContainerStarted","Data":"e64cde6988e8d049373a6e609b79a0f2e62a4d924b627bb3e04fd1f1f97ee053"} Oct 31 20:00:16.685701 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:16.685648 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" event={"ID":"d1110e52-d926-449c-809b-be04b7d1db94","Type":"ContainerStarted","Data":"db0e3b1b23f84402429a816fed82f31365d3fbafcb5813ea5d383b0db99defa0"} Oct 31 20:00:16.748329 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:16.748227 4910 scope.go:122] "RemoveContainer" containerID="1598d24b3baf87b8ade7038af4d19ee637b70a545abfb64eac668a79ed0cdb7b" Oct 31 20:00:16.748491 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:16.748411 4910 scope.go:122] "RemoveContainer" containerID="95d1db6eded59c9ba5da0f7b66054a19cdb474e695294e8443e4b3662809362a" Oct 31 20:00:16.748615 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:16.748592 4910 scope.go:122] "RemoveContainer" containerID="ee9429d64c32e0616d5fca9aa9d4ddc7e6e2950c3c17443ff17b89c91275f3a1" Oct 31 20:00:16.786957 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:16.786879 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" podStartSLOduration=2.786861455 podStartE2EDuration="2.786861455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:16.759652859 +0000 UTC m=+2451.122473231" watchObservedRunningTime="2026-10-31 20:00:16.786861455 +0000 UTC m=+2451.149681829" Oct 31 20:00:17.194495 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:17.188171 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:17.212372 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:17.212346 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:17.744938 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:17.744666 4910 generic.go:356] "Generic (PLEG): container finished" podID="f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33" containerID="3474beb7340ad76cd5f0e27fab6040b282ba828ac8bdca4182f100b53f193c28" exitCode=60 Oct 31 20:00:17.744938 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:17.744780 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6" event={"ID":"f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33","Type":"ContainerDied","Data":"3474beb7340ad76cd5f0e27fab6040b282ba828ac8bdca4182f100b53f193c28"} Oct 31 20:00:17.744938 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:17.744835 4910 scope.go:122] "RemoveContainer" containerID="fa35982c77cee87c2b28b39861f7e7ed492f8361da9f452603af156d0428a86f" Oct 31 20:00:17.762937 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:17.745404 4910 scope.go:122] "RemoveContainer" containerID="3474beb7340ad76cd5f0e27fab6040b282ba828ac8bdca4182f100b53f193c28" Oct 31 20:00:17.762937 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:17.745856 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6_odh-ai-gateway-infra(f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6" podUID="f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33" Oct 31 20:00:17.762937 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:17.760515 4910 scope.go:122] "RemoveContainer" containerID="03fcf416b47b16c4baa4d16cc2beaa463cf92264e666cb64bae0be0355ce1d86" Oct 31 20:00:17.762937 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:17.761371 4910 scope.go:122] "RemoveContainer" containerID="0bbf8ca4958847b51aedc2cf8dcfd37de48f3b57d412b9b348db2e3c6c046aa1" Oct 31 20:00:17.801454 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:17.781595 4910 scope.go:122] "RemoveContainer" containerID="c72b8791a4a67a4787f72cf1d34323d792da1eeae81f9502eb9cb485a30dc432" Oct 31 20:00:17.801454 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:17.783485 4910 scope.go:122] "RemoveContainer" containerID="0bfef8656a6612389c532f83e655329c0987b566d7f462277c304acb0d50e1ec" Oct 31 20:00:17.988998 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:17.978353 4910 generic.go:356] "Generic (PLEG): container finished" podID="e9d2cb4a-9f51-448e-bf66-af95955df697" containerID="f7b7efc4895e30026a1b8ea3e463073538f4c449221fec5d1aa025d1fb938680" exitCode=60 Oct 31 20:00:17.988998 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:17.981855 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56" event={"ID":"40440b1b-cbf1-4978-b545-fe368a4157c2","Type":"ContainerStarted","Data":"d6956b51ef93f8e99f1e7dbd7831104f5960b8da0c09b50620e60ae38202ec74"} Oct 31 20:00:17.988998 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:17.981938 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm"] Oct 31 20:00:17.988998 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:17.981978 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk" event={"ID":"99ef85db-e8f9-4826-be8c-6afdfdea13c0","Type":"ContainerStarted","Data":"aec3f1ae0e42cb76814886f6665c284bf6e5d4283de5d6cedc3fb31b82e8ba7d"} Oct 31 20:00:17.988998 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:17.982006 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm" event={"ID":"e9d2cb4a-9f51-448e-bf66-af95955df697","Type":"ContainerDied","Data":"f7b7efc4895e30026a1b8ea3e463073538f4c449221fec5d1aa025d1fb938680"} Oct 31 20:00:18.009312 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:18.009271 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9" event={"ID":"1b31bd96-7e0b-448c-b45f-825fe60427ca","Type":"ContainerStarted","Data":"45c5036c6d526b5670d96fe15c02da9ac547d757aedd218d8463e54b41e14d7d"} Oct 31 20:00:18.010743 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:18.010689 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6"] Oct 31 20:00:18.018932 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:18.017263 4910 generic.go:356] "Generic (PLEG): container finished" podID="3d30713d-b20b-47f1-9aee-087c061f7835" containerID="d6f3072dc441887d53753f5964e8f80dc364bbd4bfa35d9ffaf40c6e8c7f913e" exitCode=60 Oct 31 20:00:18.019064 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:18.018951 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s" event={"ID":"3d30713d-b20b-47f1-9aee-087c061f7835","Type":"ContainerDied","Data":"d6f3072dc441887d53753f5964e8f80dc364bbd4bfa35d9ffaf40c6e8c7f913e"} Oct 31 20:00:18.019064 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:18.019032 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:18.019786 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:18.019449 4910 scope.go:122] "RemoveContainer" containerID="d6f3072dc441887d53753f5964e8f80dc364bbd4bfa35d9ffaf40c6e8c7f913e" Oct 31 20:00:18.019902 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:18.019857 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s_odh-ai-gateway-infra(3d30713d-b20b-47f1-9aee-087c061f7835)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s" podUID="3d30713d-b20b-47f1-9aee-087c061f7835" Oct 31 20:00:18.020854 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:18.020834 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:00:18.034085 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:18.033999 4910 scope.go:122] "RemoveContainer" containerID="23baa35298e5b6767ffbfd4bb50a7ec510aea2a68e36966ca2e6d7ff27ed69e0" Oct 31 20:00:18.041238 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:18.040598 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s"] Oct 31 20:00:18.263125 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:18.263102 4910 scope.go:122] "RemoveContainer" containerID="8424c8e4a72c4f86febb235858e09dd4ff1a422d445db6f9092f52d16b8d98c8" Oct 31 20:00:18.320029 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:18.319975 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm" Oct 31 20:00:18.447505 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:18.447457 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9d2cb4a-9f51-448e-bf66-af95955df697-kube-api-access-b76hx\" (UniqueName: \"kubernetes.io/projected/e9d2cb4a-9f51-448e-bf66-af95955df697-kube-api-access-b76hx\") pod \"e9d2cb4a-9f51-448e-bf66-af95955df697\" (UID: \"e9d2cb4a-9f51-448e-bf66-af95955df697\") " Oct 31 20:00:18.447682 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:18.447598 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9d2cb4a-9f51-448e-bf66-af95955df697-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d2cb4a-9f51-448e-bf66-af95955df697-maas-api-service-ca\") pod \"e9d2cb4a-9f51-448e-bf66-af95955df697\" (UID: \"e9d2cb4a-9f51-448e-bf66-af95955df697\") " Oct 31 20:00:18.449679 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:18.449618 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d2cb4a-9f51-448e-bf66-af95955df697-maas-api-service-ca" pod "e9d2cb4a-9f51-448e-bf66-af95955df697" (UID: "e9d2cb4a-9f51-448e-bf66-af95955df697"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:18.478812 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:18.478780 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d2cb4a-9f51-448e-bf66-af95955df697-kube-api-access-b76hx" pod "e9d2cb4a-9f51-448e-bf66-af95955df697" (UID: "e9d2cb4a-9f51-448e-bf66-af95955df697"). InnerVolumeSpecName "kube-api-access-b76hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:18.550601 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:18.550574 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b76hx\" (UniqueName: \"kubernetes.io/projected/e9d2cb4a-9f51-448e-bf66-af95955df697-kube-api-access-b76hx\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:18.550737 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:18.550725 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9d2cb4a-9f51-448e-bf66-af95955df697-maas-api-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:19.032732 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:19.032664 4910 generic.go:356] "Generic (PLEG): container finished" podID="40440b1b-cbf1-4978-b545-fe368a4157c2" containerID="d6956b51ef93f8e99f1e7dbd7831104f5960b8da0c09b50620e60ae38202ec74" exitCode=60 Oct 31 20:00:19.033229 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:19.032740 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56" event={"ID":"40440b1b-cbf1-4978-b545-fe368a4157c2","Type":"ContainerDied","Data":"d6956b51ef93f8e99f1e7dbd7831104f5960b8da0c09b50620e60ae38202ec74"} Oct 31 20:00:19.033229 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:19.032805 4910 scope.go:122] "RemoveContainer" containerID="ee9429d64c32e0616d5fca9aa9d4ddc7e6e2950c3c17443ff17b89c91275f3a1" Oct 31 20:00:19.033327 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:19.033307 4910 scope.go:122] "RemoveContainer" containerID="d6956b51ef93f8e99f1e7dbd7831104f5960b8da0c09b50620e60ae38202ec74" Oct 31 20:00:19.033960 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:19.033755 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56_odh-ai-gateway-infra(40440b1b-cbf1-4978-b545-fe368a4157c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56" podUID="40440b1b-cbf1-4978-b545-fe368a4157c2" Oct 31 20:00:19.035760 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:19.035735 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5" event={"ID":"75310598-a5d8-4012-993a-fecce8281289","Type":"ContainerStarted","Data":"c989aba35bec60d64ea5548cd2cfaa1f6cd2d127ff22f4dd79b343d1a879ec07"} Oct 31 20:00:19.038300 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:19.038278 4910 generic.go:356] "Generic (PLEG): container finished" podID="99ef85db-e8f9-4826-be8c-6afdfdea13c0" containerID="aec3f1ae0e42cb76814886f6665c284bf6e5d4283de5d6cedc3fb31b82e8ba7d" exitCode=60 Oct 31 20:00:19.038419 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:19.038374 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk" event={"ID":"99ef85db-e8f9-4826-be8c-6afdfdea13c0","Type":"ContainerDied","Data":"aec3f1ae0e42cb76814886f6665c284bf6e5d4283de5d6cedc3fb31b82e8ba7d"} Oct 31 20:00:19.038822 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:19.038800 4910 scope.go:122] "RemoveContainer" containerID="aec3f1ae0e42cb76814886f6665c284bf6e5d4283de5d6cedc3fb31b82e8ba7d" Oct 31 20:00:19.039504 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:19.039379 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk_odh-ai-gateway-infra(99ef85db-e8f9-4826-be8c-6afdfdea13c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk" podUID="99ef85db-e8f9-4826-be8c-6afdfdea13c0" Oct 31 20:00:19.041005 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:19.040979 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm" event={"ID":"e9d2cb4a-9f51-448e-bf66-af95955df697","Type":"ContainerDied","Data":"1d446fd84496f5cac145e92af9208425c63aee6ad392cdb954c5af7aa0d31237"} Oct 31 20:00:19.041109 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:19.041003 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm" Oct 31 20:00:19.043787 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:19.043763 4910 generic.go:356] "Generic (PLEG): container finished" podID="1b31bd96-7e0b-448c-b45f-825fe60427ca" containerID="45c5036c6d526b5670d96fe15c02da9ac547d757aedd218d8463e54b41e14d7d" exitCode=60 Oct 31 20:00:19.043894 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:19.043838 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9" event={"ID":"1b31bd96-7e0b-448c-b45f-825fe60427ca","Type":"ContainerDied","Data":"45c5036c6d526b5670d96fe15c02da9ac547d757aedd218d8463e54b41e14d7d"} Oct 31 20:00:19.044291 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:19.044246 4910 scope.go:122] "RemoveContainer" containerID="45c5036c6d526b5670d96fe15c02da9ac547d757aedd218d8463e54b41e14d7d" Oct 31 20:00:19.044592 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:19.044567 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9_odh-ai-gateway-infra(1b31bd96-7e0b-448c-b45f-825fe60427ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9" podUID="1b31bd96-7e0b-448c-b45f-825fe60427ca" Oct 31 20:00:19.048136 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:19.048112 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn" event={"ID":"cce6c21a-41e1-4e7e-ac6d-2cc90582803a","Type":"ContainerStarted","Data":"0aac9196f70dff46d28bae87af1370039da1e5569ee76e63d05e338109b1d298"} Oct 31 20:00:19.054850 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:19.054828 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb" event={"ID":"5a336785-37c0-4fa9-a100-4872bc215b03","Type":"ContainerStarted","Data":"bbde37ea0543b857bc5cb7742258a31da6deb45c44213ac3aeed02d8bdfbed94"} Oct 31 20:00:19.055526 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:19.055504 4910 scope.go:122] "RemoveContainer" containerID="1598d24b3baf87b8ade7038af4d19ee637b70a545abfb64eac668a79ed0cdb7b" Oct 31 20:00:19.058126 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:19.058098 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c" event={"ID":"12bf7929-e3b2-406e-9095-450b003c84e9","Type":"ContainerStarted","Data":"771305ba226a4135e0f3d7f46f8bcef12200c4ae72ba67be0e4681f1c9df4441"} Oct 31 20:00:19.123389 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:19.116929 4910 scope.go:122] "RemoveContainer" containerID="f7b7efc4895e30026a1b8ea3e463073538f4c449221fec5d1aa025d1fb938680" Oct 31 20:00:19.186057 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:19.186010 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk"] Oct 31 20:00:19.626951 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:19.607571 4910 scope.go:122] "RemoveContainer" containerID="95d1db6eded59c9ba5da0f7b66054a19cdb474e695294e8443e4b3662809362a" Oct 31 20:00:20.156188 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:20.151826 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:00:20.328320 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:20.230398 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:00:20.511991 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:20.500215 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:00:20.512373 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:20.512334 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:00:20.512559 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:20.512540 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:00:20.512693 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:20.512677 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 20:00:20.533703 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:20.533662 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:00:20.547615 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:20.547562 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:00:20.549976 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:20.549940 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:00:20.580755 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:20.558963 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 20:00:20.584102 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:20.584058 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 20:00:20.607147 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:20.603583 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" podUID="c20b518f-fe19-47b6-88a6-aec10c91cf71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.244:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:00:20.607147 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:20.605887 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:00:20.740506 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:20.740455 4910 generic.go:356] "Generic (PLEG): container finished" podID="5a336785-37c0-4fa9-a100-4872bc215b03" containerID="bbde37ea0543b857bc5cb7742258a31da6deb45c44213ac3aeed02d8bdfbed94" exitCode=60 Oct 31 20:00:20.741029 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:20.740984 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb" event={"ID":"5a336785-37c0-4fa9-a100-4872bc215b03","Type":"ContainerDied","Data":"bbde37ea0543b857bc5cb7742258a31da6deb45c44213ac3aeed02d8bdfbed94"} Oct 31 20:00:20.741302 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:20.741282 4910 scope.go:122] "RemoveContainer" containerID="0bbf8ca4958847b51aedc2cf8dcfd37de48f3b57d412b9b348db2e3c6c046aa1" Oct 31 20:00:20.742134 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:20.742113 4910 scope.go:122] "RemoveContainer" containerID="bbde37ea0543b857bc5cb7742258a31da6deb45c44213ac3aeed02d8bdfbed94" Oct 31 20:00:20.742724 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:20.742699 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891280-r2lcb_odh-ai-gateway-infra(5a336785-37c0-4fa9-a100-4872bc215b03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb" podUID="5a336785-37c0-4fa9-a100-4872bc215b03" Oct 31 20:00:20.752824 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:20.752796 4910 generic.go:356] "Generic (PLEG): container finished" podID="12bf7929-e3b2-406e-9095-450b003c84e9" containerID="771305ba226a4135e0f3d7f46f8bcef12200c4ae72ba67be0e4681f1c9df4441" exitCode=60 Oct 31 20:00:20.753091 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:20.752978 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c" event={"ID":"12bf7929-e3b2-406e-9095-450b003c84e9","Type":"ContainerDied","Data":"771305ba226a4135e0f3d7f46f8bcef12200c4ae72ba67be0e4681f1c9df4441"} Oct 31 20:00:20.753582 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:20.753565 4910 scope.go:122] "RemoveContainer" containerID="771305ba226a4135e0f3d7f46f8bcef12200c4ae72ba67be0e4681f1c9df4441" Oct 31 20:00:20.758894 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:20.758268 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:00:20.808671 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:20.806077 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c_odh-ai-gateway-infra(12bf7929-e3b2-406e-9095-450b003c84e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c" podUID="12bf7929-e3b2-406e-9095-450b003c84e9" Oct 31 20:00:20.897232 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:20.894289 4910 scope.go:122] "RemoveContainer" containerID="0bfef8656a6612389c532f83e655329c0987b566d7f462277c304acb0d50e1ec" Oct 31 20:00:20.912838 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:20.911580 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 20:00:20.967003 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:20.966958 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm"] Oct 31 20:00:21.581170 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:21.581115 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:00:21.617951 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:21.610973 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" podUID="8fa2f1d6-abaa-4c73-8fdc-b91691bfb030" containerName="main" probeResult="failure" output="Get \"https://10.132.0.243:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:00:21.661178 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:21.649045 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 20:00:21.677132 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:21.677086 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 20:00:21.715476 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:21.679858 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:00:21.715476 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:21.702076 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:00:21.797935 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:21.797762 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 20:00:22.038084 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.038037 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:00:22.053763 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.053730 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 20:00:22.124963 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.120271 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 20:00:22.292498 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.105508 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" podUID="c20b518f-fe19-47b6-88a6-aec10c91cf71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.244:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:00:22.382944 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.369280 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:00:22.382944 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.370325 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 20:00:22.420446 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.418608 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s" Oct 31 20:00:22.449590 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.449550 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56"] Oct 31 20:00:22.449789 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.449671 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 20:00:22.449789 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.449712 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 20:00:22.449789 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.449743 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 20:00:22.449789 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.449772 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 20:00:22.449789 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.449787 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 20:00:22.450096 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.449817 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 20:00:22.450096 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.449854 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 20:00:22.450096 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.449886 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 20:00:22.458574 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.458547 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 20:00:22.459354 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.459081 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 20:00:22.459354 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.459134 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 20:00:22.459354 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.459168 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 20:00:22.463443 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.463424 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6" Oct 31 20:00:22.507758 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.503104 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5bbf75-29891280-msmfm"] Oct 31 20:00:22.540897 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.540606 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9"] Oct 31 20:00:22.546676 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.545278 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d30713d-b20b-47f1-9aee-087c061f7835-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d30713d-b20b-47f1-9aee-087c061f7835-maas-api-service-ca\") pod \"3d30713d-b20b-47f1-9aee-087c061f7835\" (UID: \"3d30713d-b20b-47f1-9aee-087c061f7835\") " Oct 31 20:00:22.546676 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.545369 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d30713d-b20b-47f1-9aee-087c061f7835-kube-api-access-55ch5\" (UniqueName: \"kubernetes.io/projected/3d30713d-b20b-47f1-9aee-087c061f7835-kube-api-access-55ch5\") pod \"3d30713d-b20b-47f1-9aee-087c061f7835\" (UID: \"3d30713d-b20b-47f1-9aee-087c061f7835\") " Oct 31 20:00:22.546676 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.546621 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d30713d-b20b-47f1-9aee-087c061f7835-maas-api-service-ca" pod "3d30713d-b20b-47f1-9aee-087c061f7835" (UID: "3d30713d-b20b-47f1-9aee-087c061f7835"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:22.549106 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.547730 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33-kube-api-access-xqlsm\" (UniqueName: \"kubernetes.io/projected/f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33-kube-api-access-xqlsm\") pod \"f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33\" (UID: \"f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33\") " Oct 31 20:00:22.549106 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.548738 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d30713d-b20b-47f1-9aee-087c061f7835-maas-api-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:22.600800 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.595154 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d30713d-b20b-47f1-9aee-087c061f7835-kube-api-access-55ch5" pod "3d30713d-b20b-47f1-9aee-087c061f7835" (UID: "3d30713d-b20b-47f1-9aee-087c061f7835"). InnerVolumeSpecName "kube-api-access-55ch5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:22.600800 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.598194 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33-kube-api-access-xqlsm" pod "f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33" (UID: "f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33"). InnerVolumeSpecName "kube-api-access-xqlsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:22.685345 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.651451 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33-maas-api-service-ca\") pod \"f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33\" (UID: \"f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33\") " Oct 31 20:00:22.685345 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.653395 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqlsm\" (UniqueName: \"kubernetes.io/projected/f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33-kube-api-access-xqlsm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:22.685345 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.653431 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55ch5\" (UniqueName: \"kubernetes.io/projected/3d30713d-b20b-47f1-9aee-087c061f7835-kube-api-access-55ch5\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:22.685345 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.654229 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33-maas-api-service-ca" pod "f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33" (UID: "f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:22.721060 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.720987 4910 generic.go:356] "Generic (PLEG): container finished" podID="cce6c21a-41e1-4e7e-ac6d-2cc90582803a" containerID="0aac9196f70dff46d28bae87af1370039da1e5569ee76e63d05e338109b1d298" exitCode=60 Oct 31 20:00:22.721326 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.721306 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn" event={"ID":"cce6c21a-41e1-4e7e-ac6d-2cc90582803a","Type":"ContainerDied","Data":"0aac9196f70dff46d28bae87af1370039da1e5569ee76e63d05e338109b1d298"} Oct 31 20:00:22.721461 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.721451 4910 scope.go:122] "RemoveContainer" containerID="03fcf416b47b16c4baa4d16cc2beaa463cf92264e666cb64bae0be0355ce1d86" Oct 31 20:00:22.721993 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.721977 4910 scope.go:122] "RemoveContainer" containerID="0aac9196f70dff46d28bae87af1370039da1e5569ee76e63d05e338109b1d298" Oct 31 20:00:22.722538 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:22.722512 4910 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn_odh-ai-gateway-infra(cce6c21a-41e1-4e7e-ac6d-2cc90582803a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn" podUID="cce6c21a-41e1-4e7e-ac6d-2cc90582803a" Oct 31 20:00:22.746088 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.746055 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5"] Oct 31 20:00:22.746344 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.746315 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5" podUID="75310598-a5d8-4012-993a-fecce8281289" containerName="cleanup" containerID="cri-o://c989aba35bec60d64ea5548cd2cfaa1f6cd2d127ff22f4dd79b343d1a879ec07" gracePeriod=30 Oct 31 20:00:22.746627 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.746563 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s" event={"ID":"3d30713d-b20b-47f1-9aee-087c061f7835","Type":"ContainerDied","Data":"9ca4eacfcb59120200509bf7a74c80b7838723dd3d464fedfa1e3a739e8ac94d"} Oct 31 20:00:22.749259 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.746799 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s" Oct 31 20:00:22.757379 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.757328 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33-maas-api-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:22.758401 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.757559 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6" event={"ID":"f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33","Type":"ContainerDied","Data":"3aefaef8ec7297e554aa1480f18faaada77c0cdada99a8316500f3340c5afb9d"} Oct 31 20:00:22.758401 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.757594 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6" Oct 31 20:00:22.766733 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.766708 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn"] Oct 31 20:00:22.770528 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.770511 4910 scope.go:122] "RemoveContainer" containerID="d6f3072dc441887d53753f5964e8f80dc364bbd4bfa35d9ffaf40c6e8c7f913e" Oct 31 20:00:22.774321 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.774275 4910 generic.go:356] "Generic (PLEG): container finished" podID="75310598-a5d8-4012-993a-fecce8281289" containerID="c989aba35bec60d64ea5548cd2cfaa1f6cd2d127ff22f4dd79b343d1a879ec07" exitCode=60 Oct 31 20:00:22.778849 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.778815 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5" event={"ID":"75310598-a5d8-4012-993a-fecce8281289","Type":"ContainerDied","Data":"c989aba35bec60d64ea5548cd2cfaa1f6cd2d127ff22f4dd79b343d1a879ec07"} Oct 31 20:00:22.905341 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.898697 4910 scope.go:122] "RemoveContainer" containerID="3474beb7340ad76cd5f0e27fab6040b282ba828ac8bdca4182f100b53f193c28" Oct 31 20:00:22.916011 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.915983 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c"] Oct 31 20:00:22.916879 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.916793 4910 scope.go:122] "RemoveContainer" containerID="c72b8791a4a67a4787f72cf1d34323d792da1eeae81f9502eb9cb485a30dc432" Oct 31 20:00:22.934855 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:22.934584 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb"] Oct 31 20:00:23.083510 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.079498 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56" Oct 31 20:00:23.094651 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.094442 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk" Oct 31 20:00:23.098206 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.097590 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9" Oct 31 20:00:23.104553 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.102162 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99ef85db-e8f9-4826-be8c-6afdfdea13c0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99ef85db-e8f9-4826-be8c-6afdfdea13c0-maas-api-service-ca\") pod \"99ef85db-e8f9-4826-be8c-6afdfdea13c0\" (UID: \"99ef85db-e8f9-4826-be8c-6afdfdea13c0\") " Oct 31 20:00:23.104553 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.103040 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40440b1b-cbf1-4978-b545-fe368a4157c2-kube-api-access-g6fzm\" (UniqueName: \"kubernetes.io/projected/40440b1b-cbf1-4978-b545-fe368a4157c2-kube-api-access-g6fzm\") pod \"40440b1b-cbf1-4978-b545-fe368a4157c2\" (UID: \"40440b1b-cbf1-4978-b545-fe368a4157c2\") " Oct 31 20:00:23.104553 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.103261 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40440b1b-cbf1-4978-b545-fe368a4157c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40440b1b-cbf1-4978-b545-fe368a4157c2-maas-api-service-ca\") pod \"40440b1b-cbf1-4978-b545-fe368a4157c2\" (UID: \"40440b1b-cbf1-4978-b545-fe368a4157c2\") " Oct 31 20:00:23.104553 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.103482 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99ef85db-e8f9-4826-be8c-6afdfdea13c0-kube-api-access-qpfm7\" (UniqueName: \"kubernetes.io/projected/99ef85db-e8f9-4826-be8c-6afdfdea13c0-kube-api-access-qpfm7\") pod \"99ef85db-e8f9-4826-be8c-6afdfdea13c0\" (UID: \"99ef85db-e8f9-4826-be8c-6afdfdea13c0\") " Oct 31 20:00:23.106007 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.105810 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40440b1b-cbf1-4978-b545-fe368a4157c2-maas-api-service-ca" pod "40440b1b-cbf1-4978-b545-fe368a4157c2" (UID: "40440b1b-cbf1-4978-b545-fe368a4157c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:23.106007 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.105980 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99ef85db-e8f9-4826-be8c-6afdfdea13c0-maas-api-service-ca" pod "99ef85db-e8f9-4826-be8c-6afdfdea13c0" (UID: "99ef85db-e8f9-4826-be8c-6afdfdea13c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:23.122470 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.112165 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40440b1b-cbf1-4978-b545-fe368a4157c2-kube-api-access-g6fzm" pod "40440b1b-cbf1-4978-b545-fe368a4157c2" (UID: "40440b1b-cbf1-4978-b545-fe368a4157c2"). InnerVolumeSpecName "kube-api-access-g6fzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:23.122470 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.112198 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ef85db-e8f9-4826-be8c-6afdfdea13c0-kube-api-access-qpfm7" pod "99ef85db-e8f9-4826-be8c-6afdfdea13c0" (UID: "99ef85db-e8f9-4826-be8c-6afdfdea13c0"). InnerVolumeSpecName "kube-api-access-qpfm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:23.194041 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.194011 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5" Oct 31 20:00:23.207026 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.205676 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75310598-a5d8-4012-993a-fecce8281289-kube-api-access-kmz7m\" (UniqueName: \"kubernetes.io/projected/75310598-a5d8-4012-993a-fecce8281289-kube-api-access-kmz7m\") pod \"75310598-a5d8-4012-993a-fecce8281289\" (UID: \"75310598-a5d8-4012-993a-fecce8281289\") " Oct 31 20:00:23.207026 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.205780 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75310598-a5d8-4012-993a-fecce8281289-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75310598-a5d8-4012-993a-fecce8281289-maas-api-service-ca\") pod \"75310598-a5d8-4012-993a-fecce8281289\" (UID: \"75310598-a5d8-4012-993a-fecce8281289\") " Oct 31 20:00:23.207026 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.205887 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b31bd96-7e0b-448c-b45f-825fe60427ca-kube-api-access-f5vl4\" (UniqueName: \"kubernetes.io/projected/1b31bd96-7e0b-448c-b45f-825fe60427ca-kube-api-access-f5vl4\") pod \"1b31bd96-7e0b-448c-b45f-825fe60427ca\" (UID: \"1b31bd96-7e0b-448c-b45f-825fe60427ca\") " Oct 31 20:00:23.207026 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.206049 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b31bd96-7e0b-448c-b45f-825fe60427ca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b31bd96-7e0b-448c-b45f-825fe60427ca-maas-api-service-ca\") pod \"1b31bd96-7e0b-448c-b45f-825fe60427ca\" (UID: \"1b31bd96-7e0b-448c-b45f-825fe60427ca\") " Oct 31 20:00:23.207026 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.206760 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpfm7\" (UniqueName: \"kubernetes.io/projected/99ef85db-e8f9-4826-be8c-6afdfdea13c0-kube-api-access-qpfm7\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:23.207026 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.206780 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99ef85db-e8f9-4826-be8c-6afdfdea13c0-maas-api-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:23.207026 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.206791 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6fzm\" (UniqueName: \"kubernetes.io/projected/40440b1b-cbf1-4978-b545-fe368a4157c2-kube-api-access-g6fzm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:23.207026 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.206801 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40440b1b-cbf1-4978-b545-fe368a4157c2-maas-api-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:23.207539 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.207509 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b31bd96-7e0b-448c-b45f-825fe60427ca-maas-api-service-ca" pod "1b31bd96-7e0b-448c-b45f-825fe60427ca" (UID: "1b31bd96-7e0b-448c-b45f-825fe60427ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:23.209345 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.209306 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75310598-a5d8-4012-993a-fecce8281289-maas-api-service-ca" pod "75310598-a5d8-4012-993a-fecce8281289" (UID: "75310598-a5d8-4012-993a-fecce8281289"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:23.212729 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.212667 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s"] Oct 31 20:00:23.213202 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.213145 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75310598-a5d8-4012-993a-fecce8281289-kube-api-access-kmz7m" pod "75310598-a5d8-4012-993a-fecce8281289" (UID: "75310598-a5d8-4012-993a-fecce8281289"). InnerVolumeSpecName "kube-api-access-kmz7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:23.213455 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.213426 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b31bd96-7e0b-448c-b45f-825fe60427ca-kube-api-access-f5vl4" pod "1b31bd96-7e0b-448c-b45f-825fe60427ca" (UID: "1b31bd96-7e0b-448c-b45f-825fe60427ca"). InnerVolumeSpecName "kube-api-access-f5vl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:23.228009 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.227907 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-e1d060-29891280-b886s"] Oct 31 20:00:23.307975 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.306751 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c" Oct 31 20:00:23.307975 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.307818 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmz7m\" (UniqueName: \"kubernetes.io/projected/75310598-a5d8-4012-993a-fecce8281289-kube-api-access-kmz7m\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:23.307975 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.307837 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75310598-a5d8-4012-993a-fecce8281289-maas-api-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:23.307975 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.307848 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5vl4\" (UniqueName: \"kubernetes.io/projected/1b31bd96-7e0b-448c-b45f-825fe60427ca-kube-api-access-f5vl4\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:23.307975 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.307857 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b31bd96-7e0b-448c-b45f-825fe60427ca-maas-api-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:23.308847 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.308822 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb" Oct 31 20:00:23.316058 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.316033 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6"] Oct 31 20:00:23.325081 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.325023 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891280-4r6x6"] Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.334295 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6"] Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.335980 4910 cpu_manager.go:414] "containerMap: removing container" podUID="3d30713d-b20b-47f1-9aee-087c061f7835" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336000 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d30713d-b20b-47f1-9aee-087c061f7835" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336027 4910 cpu_manager.go:414] "containerMap: removing container" podUID="99ef85db-e8f9-4826-be8c-6afdfdea13c0" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336034 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99ef85db-e8f9-4826-be8c-6afdfdea13c0" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336049 4910 cpu_manager.go:414] "containerMap: removing container" podUID="f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336058 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336080 4910 cpu_manager.go:414] "containerMap: removing container" podUID="12bf7929-e3b2-406e-9095-450b003c84e9" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336088 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12bf7929-e3b2-406e-9095-450b003c84e9" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336103 4910 cpu_manager.go:414] "containerMap: removing container" podUID="e9d2cb4a-9f51-448e-bf66-af95955df697" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336110 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d2cb4a-9f51-448e-bf66-af95955df697" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336133 4910 cpu_manager.go:414] "containerMap: removing container" podUID="12bf7929-e3b2-406e-9095-450b003c84e9" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336140 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12bf7929-e3b2-406e-9095-450b003c84e9" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336163 4910 cpu_manager.go:414] "containerMap: removing container" podUID="1b31bd96-7e0b-448c-b45f-825fe60427ca" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336169 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b31bd96-7e0b-448c-b45f-825fe60427ca" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336188 4910 cpu_manager.go:414] "containerMap: removing container" podUID="3d30713d-b20b-47f1-9aee-087c061f7835" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336194 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d30713d-b20b-47f1-9aee-087c061f7835" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336211 4910 cpu_manager.go:414] "containerMap: removing container" podUID="75310598-a5d8-4012-993a-fecce8281289" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336218 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75310598-a5d8-4012-993a-fecce8281289" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336235 4910 cpu_manager.go:414] "containerMap: removing container" podUID="40440b1b-cbf1-4978-b545-fe368a4157c2" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336241 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40440b1b-cbf1-4978-b545-fe368a4157c2" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336261 4910 cpu_manager.go:414] "containerMap: removing container" podUID="40440b1b-cbf1-4978-b545-fe368a4157c2" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336267 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40440b1b-cbf1-4978-b545-fe368a4157c2" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336287 4910 cpu_manager.go:414] "containerMap: removing container" podUID="5a336785-37c0-4fa9-a100-4872bc215b03" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336293 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a336785-37c0-4fa9-a100-4872bc215b03" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336314 4910 cpu_manager.go:414] "containerMap: removing container" podUID="99ef85db-e8f9-4826-be8c-6afdfdea13c0" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336321 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99ef85db-e8f9-4826-be8c-6afdfdea13c0" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336330 4910 cpu_manager.go:414] "containerMap: removing container" podUID="40440b1b-cbf1-4978-b545-fe368a4157c2" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336337 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40440b1b-cbf1-4978-b545-fe368a4157c2" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336365 4910 cpu_manager.go:414] "containerMap: removing container" podUID="99ef85db-e8f9-4826-be8c-6afdfdea13c0" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336372 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99ef85db-e8f9-4826-be8c-6afdfdea13c0" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336392 4910 cpu_manager.go:414] "containerMap: removing container" podUID="1b31bd96-7e0b-448c-b45f-825fe60427ca" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336398 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b31bd96-7e0b-448c-b45f-825fe60427ca" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336430 4910 cpu_manager.go:414] "containerMap: removing container" podUID="e9d2cb4a-9f51-448e-bf66-af95955df697" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336437 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d2cb4a-9f51-448e-bf66-af95955df697" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336448 4910 cpu_manager.go:414] "containerMap: removing container" podUID="f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336460 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336469 4910 cpu_manager.go:414] "containerMap: removing container" podUID="75310598-a5d8-4012-993a-fecce8281289" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336475 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75310598-a5d8-4012-993a-fecce8281289" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336506 4910 cpu_manager.go:414] "containerMap: removing container" podUID="5a336785-37c0-4fa9-a100-4872bc215b03" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336513 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a336785-37c0-4fa9-a100-4872bc215b03" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336527 4910 cpu_manager.go:414] "containerMap: removing container" podUID="5a336785-37c0-4fa9-a100-4872bc215b03" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336534 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a336785-37c0-4fa9-a100-4872bc215b03" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336559 4910 cpu_manager.go:414] "containerMap: removing container" podUID="1b31bd96-7e0b-448c-b45f-825fe60427ca" containerName="cleanup" Oct 31 20:00:23.336960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.336565 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b31bd96-7e0b-448c-b45f-825fe60427ca" containerName="cleanup" Oct 31 20:00:23.339344 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.339313 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d2cb4a-9f51-448e-bf66-af95955df697" containerName="cleanup" Oct 31 20:00:23.339344 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.339347 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33" containerName="cleanup" Oct 31 20:00:23.339528 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.339369 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="75310598-a5d8-4012-993a-fecce8281289" containerName="cleanup" Oct 31 20:00:23.339528 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.339380 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d2cb4a-9f51-448e-bf66-af95955df697" containerName="cleanup" Oct 31 20:00:23.339528 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.339400 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a336785-37c0-4fa9-a100-4872bc215b03" containerName="cleanup" Oct 31 20:00:23.339528 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.339408 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a336785-37c0-4fa9-a100-4872bc215b03" containerName="cleanup" Oct 31 20:00:23.339528 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.339425 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d30713d-b20b-47f1-9aee-087c061f7835" containerName="cleanup" Oct 31 20:00:23.339528 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.339438 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b31bd96-7e0b-448c-b45f-825fe60427ca" containerName="cleanup" Oct 31 20:00:23.339528 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.339455 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="75310598-a5d8-4012-993a-fecce8281289" containerName="cleanup" Oct 31 20:00:23.339528 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.339464 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33" containerName="cleanup" Oct 31 20:00:23.339528 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.339492 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="99ef85db-e8f9-4826-be8c-6afdfdea13c0" containerName="cleanup" Oct 31 20:00:23.339528 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.339502 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33" containerName="cleanup" Oct 31 20:00:23.339528 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.339520 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="12bf7929-e3b2-406e-9095-450b003c84e9" containerName="cleanup" Oct 31 20:00:23.340182 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.339537 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d2cb4a-9f51-448e-bf66-af95955df697" containerName="cleanup" Oct 31 20:00:23.340182 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.339561 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="12bf7929-e3b2-406e-9095-450b003c84e9" containerName="cleanup" Oct 31 20:00:23.340182 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.339580 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d30713d-b20b-47f1-9aee-087c061f7835" containerName="cleanup" Oct 31 20:00:23.340182 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.339593 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="75310598-a5d8-4012-993a-fecce8281289" containerName="cleanup" Oct 31 20:00:23.340182 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.339610 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="40440b1b-cbf1-4978-b545-fe368a4157c2" containerName="cleanup" Oct 31 20:00:23.340182 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.339619 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="40440b1b-cbf1-4978-b545-fe368a4157c2" containerName="cleanup" Oct 31 20:00:23.340182 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.339637 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="99ef85db-e8f9-4826-be8c-6afdfdea13c0" containerName="cleanup" Oct 31 20:00:23.340182 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.339652 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="99ef85db-e8f9-4826-be8c-6afdfdea13c0" containerName="cleanup" Oct 31 20:00:23.340182 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.339669 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b31bd96-7e0b-448c-b45f-825fe60427ca" containerName="cleanup" Oct 31 20:00:23.340182 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.339679 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="12bf7929-e3b2-406e-9095-450b003c84e9" containerName="cleanup" Oct 31 20:00:23.347602 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.347549 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6" Oct 31 20:00:23.365808 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.365782 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:00:23.373080 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.368118 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6"] Oct 31 20:00:23.409174 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.409140 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12bf7929-e3b2-406e-9095-450b003c84e9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12bf7929-e3b2-406e-9095-450b003c84e9-maas-api-service-ca\") pod \"12bf7929-e3b2-406e-9095-450b003c84e9\" (UID: \"12bf7929-e3b2-406e-9095-450b003c84e9\") " Oct 31 20:00:23.411833 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.409201 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a336785-37c0-4fa9-a100-4872bc215b03-kube-api-access-6hzvz\" (UniqueName: \"kubernetes.io/projected/5a336785-37c0-4fa9-a100-4872bc215b03-kube-api-access-6hzvz\") pod \"5a336785-37c0-4fa9-a100-4872bc215b03\" (UID: \"5a336785-37c0-4fa9-a100-4872bc215b03\") " Oct 31 20:00:23.411833 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.409411 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12bf7929-e3b2-406e-9095-450b003c84e9-kube-api-access-jh26l\" (UniqueName: \"kubernetes.io/projected/12bf7929-e3b2-406e-9095-450b003c84e9-kube-api-access-jh26l\") pod \"12bf7929-e3b2-406e-9095-450b003c84e9\" (UID: \"12bf7929-e3b2-406e-9095-450b003c84e9\") " Oct 31 20:00:23.411833 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.409472 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a336785-37c0-4fa9-a100-4872bc215b03-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a336785-37c0-4fa9-a100-4872bc215b03-maas-api-service-ca\") pod \"5a336785-37c0-4fa9-a100-4872bc215b03\" (UID: \"5a336785-37c0-4fa9-a100-4872bc215b03\") " Oct 31 20:00:23.411833 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.409544 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12bf7929-e3b2-406e-9095-450b003c84e9-maas-api-service-ca" pod "12bf7929-e3b2-406e-9095-450b003c84e9" (UID: "12bf7929-e3b2-406e-9095-450b003c84e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:23.411833 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.409870 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7aaf216-7be2-4ed9-932d-b5971516ffda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6\" (UID: \"c7aaf216-7be2-4ed9-932d-b5971516ffda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6" Oct 31 20:00:23.411833 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.410122 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a336785-37c0-4fa9-a100-4872bc215b03-maas-api-service-ca" pod "5a336785-37c0-4fa9-a100-4872bc215b03" (UID: "5a336785-37c0-4fa9-a100-4872bc215b03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:23.411833 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.410653 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7aaf216-7be2-4ed9-932d-b5971516ffda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6\" (UID: \"c7aaf216-7be2-4ed9-932d-b5971516ffda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6" Oct 31 20:00:23.411833 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.410891 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plmvj\" (UniqueName: \"kubernetes.io/projected/c7aaf216-7be2-4ed9-932d-b5971516ffda-kube-api-access-plmvj\") pod \"maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6\" (UID: \"c7aaf216-7be2-4ed9-932d-b5971516ffda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6" Oct 31 20:00:23.411833 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.411102 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a336785-37c0-4fa9-a100-4872bc215b03-maas-api-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:23.411833 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.411121 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12bf7929-e3b2-406e-9095-450b003c84e9-maas-api-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:23.412481 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.412320 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12bf7929-e3b2-406e-9095-450b003c84e9-kube-api-access-jh26l" pod "12bf7929-e3b2-406e-9095-450b003c84e9" (UID: "12bf7929-e3b2-406e-9095-450b003c84e9"). InnerVolumeSpecName "kube-api-access-jh26l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:23.415407 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.415379 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a336785-37c0-4fa9-a100-4872bc215b03-kube-api-access-6hzvz" pod "5a336785-37c0-4fa9-a100-4872bc215b03" (UID: "5a336785-37c0-4fa9-a100-4872bc215b03"). InnerVolumeSpecName "kube-api-access-6hzvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:23.534476 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.531190 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7aaf216-7be2-4ed9-932d-b5971516ffda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6\" (UID: \"c7aaf216-7be2-4ed9-932d-b5971516ffda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6" Oct 31 20:00:23.594325 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.577083 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plmvj\" (UniqueName: \"kubernetes.io/projected/c7aaf216-7be2-4ed9-932d-b5971516ffda-kube-api-access-plmvj\") pod \"maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6\" (UID: \"c7aaf216-7be2-4ed9-932d-b5971516ffda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6" Oct 31 20:00:23.594325 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.577280 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7aaf216-7be2-4ed9-932d-b5971516ffda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6\" (UID: \"c7aaf216-7be2-4ed9-932d-b5971516ffda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6" Oct 31 20:00:23.594325 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.577428 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hzvz\" (UniqueName: \"kubernetes.io/projected/5a336785-37c0-4fa9-a100-4872bc215b03-kube-api-access-6hzvz\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:23.594325 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.577447 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh26l\" (UniqueName: \"kubernetes.io/projected/12bf7929-e3b2-406e-9095-450b003c84e9-kube-api-access-jh26l\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:23.594325 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.578321 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7aaf216-7be2-4ed9-932d-b5971516ffda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6\" (UID: \"c7aaf216-7be2-4ed9-932d-b5971516ffda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6" Oct 31 20:00:23.632997 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.632962 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7aaf216-7be2-4ed9-932d-b5971516ffda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6\" (UID: \"c7aaf216-7be2-4ed9-932d-b5971516ffda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6" Oct 31 20:00:23.639506 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.639442 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plmvj\" (UniqueName: \"kubernetes.io/projected/c7aaf216-7be2-4ed9-932d-b5971516ffda-kube-api-access-plmvj\") pod \"maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6\" (UID: \"c7aaf216-7be2-4ed9-932d-b5971516ffda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6" Oct 31 20:00:23.676059 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.676026 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6" Oct 31 20:00:23.771070 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.770982 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d30713d-b20b-47f1-9aee-087c061f7835" path="/var/lib/kubelet/pods/3d30713d-b20b-47f1-9aee-087c061f7835/volumes" Oct 31 20:00:23.771813 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.771788 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9d2cb4a-9f51-448e-bf66-af95955df697" path="/var/lib/kubelet/pods/e9d2cb4a-9f51-448e-bf66-af95955df697/volumes" Oct 31 20:00:23.772353 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.772332 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33" path="/var/lib/kubelet/pods/f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33/volumes" Oct 31 20:00:23.806988 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.806489 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5" event={"ID":"75310598-a5d8-4012-993a-fecce8281289","Type":"ContainerDied","Data":"db4da3efca8054104ae0f4227651ec5d0a758e392951f0a352e038ebbe0c012b"} Oct 31 20:00:23.806988 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.806546 4910 scope.go:122] "RemoveContainer" containerID="c989aba35bec60d64ea5548cd2cfaa1f6cd2d127ff22f4dd79b343d1a879ec07" Oct 31 20:00:23.806988 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.806675 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5" Oct 31 20:00:23.810635 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.810069 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk" event={"ID":"99ef85db-e8f9-4826-be8c-6afdfdea13c0","Type":"ContainerDied","Data":"b4bac5b7ec0c5e6cbc6e1288b5ccc189a9e24446f9f9fc33b747642b9fcbb968"} Oct 31 20:00:23.810635 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.810182 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk" Oct 31 20:00:23.815825 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.815760 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9" event={"ID":"1b31bd96-7e0b-448c-b45f-825fe60427ca","Type":"ContainerDied","Data":"0a970d3f58b290fcb251cdecf8ca2069a80856bf170db92bf79157495172eeda"} Oct 31 20:00:23.816630 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.815727 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9" Oct 31 20:00:23.823456 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.823168 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb" event={"ID":"5a336785-37c0-4fa9-a100-4872bc215b03","Type":"ContainerDied","Data":"87a31f0ae427f987d3016d5cebc3ae8710e5896b6dcedc6ef24ac221226b19d1"} Oct 31 20:00:23.823456 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.823271 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb" Oct 31 20:00:23.826790 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.826066 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c" event={"ID":"12bf7929-e3b2-406e-9095-450b003c84e9","Type":"ContainerDied","Data":"07d79bffdae8bbb19185b5941601b7dd1502999ff0cbc227c607cae62a3a769b"} Oct 31 20:00:23.826790 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.826143 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c" Oct 31 20:00:23.828620 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.828545 4910 scope.go:122] "RemoveContainer" containerID="aec3f1ae0e42cb76814886f6665c284bf6e5d4283de5d6cedc3fb31b82e8ba7d" Oct 31 20:00:23.832063 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.831374 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56" event={"ID":"40440b1b-cbf1-4978-b545-fe368a4157c2","Type":"ContainerDied","Data":"4a284eb1eee4e76b6ed73bc8447d04f2c30da320ac810db19295864daf5ad1c3"} Oct 31 20:00:23.832063 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.831470 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56" Oct 31 20:00:23.854454 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.854435 4910 scope.go:122] "RemoveContainer" containerID="45c5036c6d526b5670d96fe15c02da9ac547d757aedd218d8463e54b41e14d7d" Oct 31 20:00:23.871141 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.871108 4910 scope.go:122] "RemoveContainer" containerID="bbde37ea0543b857bc5cb7742258a31da6deb45c44213ac3aeed02d8bdfbed94" Oct 31 20:00:23.872655 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.872488 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9"] Oct 31 20:00:23.891036 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.890834 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae4ce2-29891280-pbql9"] Oct 31 20:00:23.894724 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.894688 4910 scope.go:122] "RemoveContainer" containerID="771305ba226a4135e0f3d7f46f8bcef12200c4ae72ba67be0e4681f1c9df4441" Oct 31 20:00:23.908062 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.908032 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk"] Oct 31 20:00:23.910865 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.910745 4910 scope.go:122] "RemoveContainer" containerID="d6956b51ef93f8e99f1e7dbd7831104f5960b8da0c09b50620e60ae38202ec74" Oct 31 20:00:23.929745 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.929716 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-cd9729-29891280-v2cxk"] Oct 31 20:00:23.946855 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.946785 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5"] Oct 31 20:00:23.960233 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:23.960199 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-26cd59-29891280-w9qb5"] Oct 31 20:00:24.006958 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:24.006598 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56"] Oct 31 20:00:24.012996 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:24.012972 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6"] Oct 31 20:00:24.027169 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:24.027129 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4bac5a-29891280-8hf56"] Oct 31 20:00:24.056287 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:24.056240 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c"] Oct 31 20:00:24.065035 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:24.065006 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-c55cf1-29891280-tc28c"] Oct 31 20:00:24.073413 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:24.073389 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb"] Oct 31 20:00:24.080712 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:24.080684 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891280-r2lcb"] Oct 31 20:00:24.141720 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:24.141687 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn" Oct 31 20:00:24.354802 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:24.354767 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce6c21a-41e1-4e7e-ac6d-2cc90582803a-kube-api-access-w9tl6\" (UniqueName: \"kubernetes.io/projected/cce6c21a-41e1-4e7e-ac6d-2cc90582803a-kube-api-access-w9tl6\") pod \"cce6c21a-41e1-4e7e-ac6d-2cc90582803a\" (UID: \"cce6c21a-41e1-4e7e-ac6d-2cc90582803a\") " Oct 31 20:00:24.373937 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:24.373877 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cce6c21a-41e1-4e7e-ac6d-2cc90582803a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce6c21a-41e1-4e7e-ac6d-2cc90582803a-maas-api-service-ca\") pod \"cce6c21a-41e1-4e7e-ac6d-2cc90582803a\" (UID: \"cce6c21a-41e1-4e7e-ac6d-2cc90582803a\") " Oct 31 20:00:24.386490 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:24.386430 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce6c21a-41e1-4e7e-ac6d-2cc90582803a-kube-api-access-w9tl6" pod "cce6c21a-41e1-4e7e-ac6d-2cc90582803a" (UID: "cce6c21a-41e1-4e7e-ac6d-2cc90582803a"). InnerVolumeSpecName "kube-api-access-w9tl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:24.425935 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:24.404379 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce6c21a-41e1-4e7e-ac6d-2cc90582803a-maas-api-service-ca" pod "cce6c21a-41e1-4e7e-ac6d-2cc90582803a" (UID: "cce6c21a-41e1-4e7e-ac6d-2cc90582803a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:24.485310 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:24.479687 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9tl6\" (UniqueName: \"kubernetes.io/projected/cce6c21a-41e1-4e7e-ac6d-2cc90582803a-kube-api-access-w9tl6\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:24.485568 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:24.485536 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cce6c21a-41e1-4e7e-ac6d-2cc90582803a-maas-api-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:24.861240 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:24.861202 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn" Oct 31 20:00:24.861660 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:24.861201 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn" event={"ID":"cce6c21a-41e1-4e7e-ac6d-2cc90582803a","Type":"ContainerDied","Data":"0eac30d61eb3c5fc5af9e9021b357bce9485c17352e47c46a28f9a3cc5375def"} Oct 31 20:00:24.861731 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:24.861691 4910 scope.go:122] "RemoveContainer" containerID="0aac9196f70dff46d28bae87af1370039da1e5569ee76e63d05e338109b1d298" Oct 31 20:00:24.865496 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:24.865468 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6" event={"ID":"c7aaf216-7be2-4ed9-932d-b5971516ffda","Type":"ContainerStarted","Data":"7c55abbaca6c93f4da76038a734de400173ec59c2dd17ea756657c7fa76dab1f"} Oct 31 20:00:24.865703 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:24.865683 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6" event={"ID":"c7aaf216-7be2-4ed9-932d-b5971516ffda","Type":"ContainerStarted","Data":"1a8e6d0224c77e10435c878b548e53d6aa5a7ad44d144453db052cd2e2868ff8"} Oct 31 20:00:24.890488 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:24.890389 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6" podStartSLOduration=1.89037465 podStartE2EDuration="1.89037465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:24.886420487 +0000 UTC m=+2459.249240861" watchObservedRunningTime="2026-10-31 20:00:24.89037465 +0000 UTC m=+2459.253195022" Oct 31 20:00:24.907645 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:24.907617 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn"] Oct 31 20:00:24.916679 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:24.916651 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891280-5ctgn"] Oct 31 20:00:25.759645 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:25.759608 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12bf7929-e3b2-406e-9095-450b003c84e9" path="/var/lib/kubelet/pods/12bf7929-e3b2-406e-9095-450b003c84e9/volumes" Oct 31 20:00:25.760260 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:25.760242 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b31bd96-7e0b-448c-b45f-825fe60427ca" path="/var/lib/kubelet/pods/1b31bd96-7e0b-448c-b45f-825fe60427ca/volumes" Oct 31 20:00:25.760735 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:25.760719 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40440b1b-cbf1-4978-b545-fe368a4157c2" path="/var/lib/kubelet/pods/40440b1b-cbf1-4978-b545-fe368a4157c2/volumes" Oct 31 20:00:25.761617 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:25.761597 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a336785-37c0-4fa9-a100-4872bc215b03" path="/var/lib/kubelet/pods/5a336785-37c0-4fa9-a100-4872bc215b03/volumes" Oct 31 20:00:25.762126 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:25.762111 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75310598-a5d8-4012-993a-fecce8281289" path="/var/lib/kubelet/pods/75310598-a5d8-4012-993a-fecce8281289/volumes" Oct 31 20:00:25.762585 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:25.762571 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99ef85db-e8f9-4826-be8c-6afdfdea13c0" path="/var/lib/kubelet/pods/99ef85db-e8f9-4826-be8c-6afdfdea13c0/volumes" Oct 31 20:00:25.763354 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:25.763339 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cce6c21a-41e1-4e7e-ac6d-2cc90582803a" path="/var/lib/kubelet/pods/cce6c21a-41e1-4e7e-ac6d-2cc90582803a/volumes" Oct 31 20:00:25.880695 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:25.880651 4910 generic.go:356] "Generic (PLEG): container finished" podID="c7aaf216-7be2-4ed9-932d-b5971516ffda" containerID="7c55abbaca6c93f4da76038a734de400173ec59c2dd17ea756657c7fa76dab1f" exitCode=0 Oct 31 20:00:25.881221 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:25.880885 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6" event={"ID":"c7aaf216-7be2-4ed9-932d-b5971516ffda","Type":"ContainerDied","Data":"7c55abbaca6c93f4da76038a734de400173ec59c2dd17ea756657c7fa76dab1f"} Oct 31 20:00:27.089567 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:27.089537 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6" Oct 31 20:00:27.157496 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:27.157410 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7aaf216-7be2-4ed9-932d-b5971516ffda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7aaf216-7be2-4ed9-932d-b5971516ffda-maas-api-openshift-ca\") pod \"c7aaf216-7be2-4ed9-932d-b5971516ffda\" (UID: \"c7aaf216-7be2-4ed9-932d-b5971516ffda\") " Oct 31 20:00:27.157669 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:27.157646 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7aaf216-7be2-4ed9-932d-b5971516ffda-kube-api-access-plmvj\" (UniqueName: \"kubernetes.io/projected/c7aaf216-7be2-4ed9-932d-b5971516ffda-kube-api-access-plmvj\") pod \"c7aaf216-7be2-4ed9-932d-b5971516ffda\" (UID: \"c7aaf216-7be2-4ed9-932d-b5971516ffda\") " Oct 31 20:00:27.157834 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:27.157806 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7aaf216-7be2-4ed9-932d-b5971516ffda-maas-api-openshift-ca" pod "c7aaf216-7be2-4ed9-932d-b5971516ffda" (UID: "c7aaf216-7be2-4ed9-932d-b5971516ffda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:27.158256 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:27.158150 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7aaf216-7be2-4ed9-932d-b5971516ffda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7aaf216-7be2-4ed9-932d-b5971516ffda-maas-api-xks-ca\") pod \"c7aaf216-7be2-4ed9-932d-b5971516ffda\" (UID: \"c7aaf216-7be2-4ed9-932d-b5971516ffda\") " Oct 31 20:00:27.159014 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:27.158994 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7aaf216-7be2-4ed9-932d-b5971516ffda-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:27.180342 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:27.173479 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7aaf216-7be2-4ed9-932d-b5971516ffda-maas-api-xks-ca" pod "c7aaf216-7be2-4ed9-932d-b5971516ffda" (UID: "c7aaf216-7be2-4ed9-932d-b5971516ffda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:00:27.185456 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:27.185424 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7aaf216-7be2-4ed9-932d-b5971516ffda-kube-api-access-plmvj" pod "c7aaf216-7be2-4ed9-932d-b5971516ffda" (UID: "c7aaf216-7be2-4ed9-932d-b5971516ffda"). InnerVolumeSpecName "kube-api-access-plmvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:27.260953 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:27.260905 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7aaf216-7be2-4ed9-932d-b5971516ffda-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:27.260953 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:27.260952 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plmvj\" (UniqueName: \"kubernetes.io/projected/c7aaf216-7be2-4ed9-932d-b5971516ffda-kube-api-access-plmvj\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:27.964932 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:27.964600 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6" event={"ID":"c7aaf216-7be2-4ed9-932d-b5971516ffda","Type":"ContainerDied","Data":"1a8e6d0224c77e10435c878b548e53d6aa5a7ad44d144453db052cd2e2868ff8"} Oct 31 20:00:27.964932 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:27.964671 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a8e6d0224c77e10435c878b548e53d6aa5a7ad44d144453db052cd2e2868ff8" Oct 31 20:00:27.964932 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:27.964756 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6" Oct 31 20:00:29.029969 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:29.029924 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bcc1e0f-sl8bc"] Oct 31 20:00:29.034594 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:29.034559 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bcc1e0f-sl8bc"] Oct 31 20:00:29.757489 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:29.757453 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="121d4267-2283-43ab-8181-dd231d3350bb" path="/var/lib/kubelet/pods/121d4267-2283-43ab-8181-dd231d3350bb/volumes" Oct 31 20:00:35.959543 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.959434 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59"] Oct 31 20:00:35.960405 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.960379 4910 cpu_manager.go:414] "containerMap: removing container" podUID="3d30713d-b20b-47f1-9aee-087c061f7835" containerName="cleanup" Oct 31 20:00:35.960405 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.960403 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d30713d-b20b-47f1-9aee-087c061f7835" containerName="cleanup" Oct 31 20:00:35.960610 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.960450 4910 cpu_manager.go:414] "containerMap: removing container" podUID="cce6c21a-41e1-4e7e-ac6d-2cc90582803a" containerName="cleanup" Oct 31 20:00:35.960610 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.960459 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce6c21a-41e1-4e7e-ac6d-2cc90582803a" containerName="cleanup" Oct 31 20:00:35.960610 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.960469 4910 cpu_manager.go:414] "containerMap: removing container" podUID="e9d2cb4a-9f51-448e-bf66-af95955df697" containerName="cleanup" Oct 31 20:00:35.960610 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.960477 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d2cb4a-9f51-448e-bf66-af95955df697" containerName="cleanup" Oct 31 20:00:35.960610 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.960490 4910 cpu_manager.go:414] "containerMap: removing container" podUID="f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33" containerName="cleanup" Oct 31 20:00:35.960610 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.960496 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9138d1d-ce67-4ecb-8afd-73d0a9ea8e33" containerName="cleanup" Oct 31 20:00:35.960610 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.960507 4910 cpu_manager.go:414] "containerMap: removing container" podUID="75310598-a5d8-4012-993a-fecce8281289" containerName="cleanup" Oct 31 20:00:35.960610 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.960515 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75310598-a5d8-4012-993a-fecce8281289" containerName="cleanup" Oct 31 20:00:35.960610 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.960529 4910 cpu_manager.go:414] "containerMap: removing container" podUID="cce6c21a-41e1-4e7e-ac6d-2cc90582803a" containerName="cleanup" Oct 31 20:00:35.960610 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.960536 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce6c21a-41e1-4e7e-ac6d-2cc90582803a" containerName="cleanup" Oct 31 20:00:35.960610 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.960585 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c7aaf216-7be2-4ed9-932d-b5971516ffda" containerName="revoke-keys" Oct 31 20:00:35.960610 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.960593 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7aaf216-7be2-4ed9-932d-b5971516ffda" containerName="revoke-keys" Oct 31 20:00:35.960610 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.960614 4910 cpu_manager.go:414] "containerMap: removing container" podUID="12bf7929-e3b2-406e-9095-450b003c84e9" containerName="cleanup" Oct 31 20:00:35.961275 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.960622 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12bf7929-e3b2-406e-9095-450b003c84e9" containerName="cleanup" Oct 31 20:00:35.961275 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.960810 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce6c21a-41e1-4e7e-ac6d-2cc90582803a" containerName="cleanup" Oct 31 20:00:35.961275 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.960827 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce6c21a-41e1-4e7e-ac6d-2cc90582803a" containerName="cleanup" Oct 31 20:00:35.961275 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.960840 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a336785-37c0-4fa9-a100-4872bc215b03" containerName="cleanup" Oct 31 20:00:35.961275 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.960852 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b31bd96-7e0b-448c-b45f-825fe60427ca" containerName="cleanup" Oct 31 20:00:35.961275 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.960866 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce6c21a-41e1-4e7e-ac6d-2cc90582803a" containerName="cleanup" Oct 31 20:00:35.961275 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.960877 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7aaf216-7be2-4ed9-932d-b5971516ffda" containerName="revoke-keys" Oct 31 20:00:35.961275 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.960890 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d30713d-b20b-47f1-9aee-087c061f7835" containerName="cleanup" Oct 31 20:00:35.961275 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.960900 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="40440b1b-cbf1-4978-b545-fe368a4157c2" containerName="cleanup" Oct 31 20:00:35.966958 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.966904 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:35.972530 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.972438 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-cd9729-openshift-default-dockercfg-4d2qm\"" Oct 31 20:00:35.979512 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:35.979454 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59"] Oct 31 20:00:36.166600 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.164883 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzrnb\" (UniqueName: \"kubernetes.io/projected/55d3a124-18b2-4198-b3ee-7ae4180538f8-kube-api-access-zzrnb\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.166600 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.164971 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-workload-socket\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.166600 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.165010 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-token\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.166600 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.165047 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-podinfo\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.172045 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.172003 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-credential-socket\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.172207 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.172053 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-workload-certs\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.172207 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.172133 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-data\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.172207 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.172183 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55d3a124-18b2-4198-b3ee-7ae4180538f8-istiod-ca-cert\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.172372 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.172290 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-envoy\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.273641 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.273553 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-workload-socket\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.273641 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.273599 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-token\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.273641 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.273628 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-podinfo\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.273875 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.273839 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-credential-socket\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.273875 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.273865 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-workload-certs\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.273964 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.273946 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-data\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.274007 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.273995 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55d3a124-18b2-4198-b3ee-7ae4180538f8-istiod-ca-cert\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.274049 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.274018 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-workload-socket\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.274162 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.274139 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-envoy\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.274267 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.274191 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-workload-certs\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.274352 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.274265 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-credential-socket\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.274352 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.274274 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzrnb\" (UniqueName: \"kubernetes.io/projected/55d3a124-18b2-4198-b3ee-7ae4180538f8-kube-api-access-zzrnb\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.274352 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.274300 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-data\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.274708 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.274689 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55d3a124-18b2-4198-b3ee-7ae4180538f8-istiod-ca-cert\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.276402 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.276380 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-envoy\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.276504 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.276435 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-podinfo\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.286887 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.286865 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-token\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.287139 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.287117 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzrnb\" (UniqueName: \"kubernetes.io/projected/55d3a124-18b2-4198-b3ee-7ae4180538f8-kube-api-access-zzrnb\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.291813 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.291791 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:36.582679 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:00:36.582645 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d3a124_18b2_4198_b3ee_7ae4180538f8.slice/crio-822c32ae02faf3582d48323ec2c05b19ae61ddb863487547450f54dc77933a02 WatchSource:0}: Error finding container 822c32ae02faf3582d48323ec2c05b19ae61ddb863487547450f54dc77933a02: Status 404 returned error can't find the container with id 822c32ae02faf3582d48323ec2c05b19ae61ddb863487547450f54dc77933a02 Oct 31 20:00:36.583182 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.583157 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59"] Oct 31 20:00:36.585657 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.585621 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:00:36.585753 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.585691 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:00:36.585753 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:36.585733 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:00:37.252831 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:37.252787 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" event={"ID":"55d3a124-18b2-4198-b3ee-7ae4180538f8","Type":"ContainerStarted","Data":"18ee071b9c433b066b559695ce61d7ba4ce7f89cbfea30442aa2cba6a19797fd"} Oct 31 20:00:37.253410 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:37.252862 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" event={"ID":"55d3a124-18b2-4198-b3ee-7ae4180538f8","Type":"ContainerStarted","Data":"822c32ae02faf3582d48323ec2c05b19ae61ddb863487547450f54dc77933a02"} Oct 31 20:00:37.292381 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:37.292343 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:37.294708 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:37.294046 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 31 20:00:37.294708 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:37.294104 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" podUID="55d3a124-18b2-4198-b3ee-7ae4180538f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 31 20:00:37.367385 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:37.367319 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" podStartSLOduration=2.367298718 podStartE2EDuration="2.367298718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:37.34783867 +0000 UTC m=+2471.710659044" watchObservedRunningTime="2026-10-31 20:00:37.367298718 +0000 UTC m=+2471.730119093" Oct 31 20:00:38.325802 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:38.325766 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 31 20:00:38.346457 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:38.346413 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" podUID="55d3a124-18b2-4198-b3ee-7ae4180538f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 31 20:00:39.296935 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:39.295375 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 31 20:00:39.296935 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:39.295448 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" podUID="55d3a124-18b2-4198-b3ee-7ae4180538f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 31 20:00:40.292783 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:40.292743 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 31 20:00:40.293396 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:40.293360 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" podUID="55d3a124-18b2-4198-b3ee-7ae4180538f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 31 20:00:40.786933 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:40.785899 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47"] Oct 31 20:00:40.786933 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:40.786619 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" podUID="13291a84-bbac-4efb-be09-e511060c6fe9" containerName="maas-api" containerID="cri-o://b83655f027f4f20c35f02c3c8f588d3f4102896f214d281668cf4fcc6544738b" gracePeriod=30 Oct 31 20:00:41.117939 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:41.108968 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e63e490b-p2ljk"] Oct 31 20:00:41.180942 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:41.178181 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e63e490b-p2ljk"] Oct 31 20:00:41.309936 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:41.308552 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 31 20:00:41.309936 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:41.308613 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" podUID="55d3a124-18b2-4198-b3ee-7ae4180538f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 31 20:00:41.324938 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:41.323067 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6"] Oct 31 20:00:41.380009 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:41.374241 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-01afb5-m6dm6"] Oct 31 20:00:41.565235 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:41.564569 4910 generic.go:356] "Generic (PLEG): container finished" podID="13291a84-bbac-4efb-be09-e511060c6fe9" containerID="b83655f027f4f20c35f02c3c8f588d3f4102896f214d281668cf4fcc6544738b" exitCode=0 Oct 31 20:00:41.565235 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:41.564871 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" event={"ID":"13291a84-bbac-4efb-be09-e511060c6fe9","Type":"ContainerDied","Data":"b83655f027f4f20c35f02c3c8f588d3f4102896f214d281668cf4fcc6544738b"} Oct 31 20:00:41.837933 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:41.836285 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" podUID="13291a84-bbac-4efb-be09-e511060c6fe9" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.242:8443/health\": dial tcp 10.132.0.242:8443: connect: connection refused" Oct 31 20:00:41.908633 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:41.907145 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="087fb9af-3c00-4e5c-bdba-9f485e3be7ec" path="/var/lib/kubelet/pods/087fb9af-3c00-4e5c-bdba-9f485e3be7ec/volumes" Oct 31 20:00:41.908633 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:41.907796 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7aaf216-7be2-4ed9-932d-b5971516ffda" path="/var/lib/kubelet/pods/c7aaf216-7be2-4ed9-932d-b5971516ffda/volumes" Oct 31 20:00:42.106686 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:42.106606 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" Oct 31 20:00:42.201655 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:42.201619 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13291a84-bbac-4efb-be09-e511060c6fe9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13291a84-bbac-4efb-be09-e511060c6fe9-maas-api-tls\") pod \"13291a84-bbac-4efb-be09-e511060c6fe9\" (UID: \"13291a84-bbac-4efb-be09-e511060c6fe9\") " Oct 31 20:00:42.201836 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:42.201777 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13291a84-bbac-4efb-be09-e511060c6fe9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/13291a84-bbac-4efb-be09-e511060c6fe9-metrics-certs\") pod \"13291a84-bbac-4efb-be09-e511060c6fe9\" (UID: \"13291a84-bbac-4efb-be09-e511060c6fe9\") " Oct 31 20:00:42.202064 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:42.202045 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13291a84-bbac-4efb-be09-e511060c6fe9-kube-api-access-d4s6j\" (UniqueName: \"kubernetes.io/projected/13291a84-bbac-4efb-be09-e511060c6fe9-kube-api-access-d4s6j\") pod \"13291a84-bbac-4efb-be09-e511060c6fe9\" (UID: \"13291a84-bbac-4efb-be09-e511060c6fe9\") " Oct 31 20:00:42.202179 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:42.202147 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13291a84-bbac-4efb-be09-e511060c6fe9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13291a84-bbac-4efb-be09-e511060c6fe9-openshift-service-ca\") pod \"13291a84-bbac-4efb-be09-e511060c6fe9\" (UID: \"13291a84-bbac-4efb-be09-e511060c6fe9\") " Oct 31 20:00:42.202810 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:42.202784 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13291a84-bbac-4efb-be09-e511060c6fe9-openshift-service-ca" pod "13291a84-bbac-4efb-be09-e511060c6fe9" (UID: "13291a84-bbac-4efb-be09-e511060c6fe9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:42.202977 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:42.202954 4910 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13291a84-bbac-4efb-be09-e511060c6fe9-openshift-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:42.204174 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:42.204148 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13291a84-bbac-4efb-be09-e511060c6fe9-kube-api-access-d4s6j" pod "13291a84-bbac-4efb-be09-e511060c6fe9" (UID: "13291a84-bbac-4efb-be09-e511060c6fe9"). InnerVolumeSpecName "kube-api-access-d4s6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:42.204440 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:42.204422 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13291a84-bbac-4efb-be09-e511060c6fe9-metrics-certs" pod "13291a84-bbac-4efb-be09-e511060c6fe9" (UID: "13291a84-bbac-4efb-be09-e511060c6fe9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:00:42.204501 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:42.204453 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13291a84-bbac-4efb-be09-e511060c6fe9-maas-api-tls" pod "13291a84-bbac-4efb-be09-e511060c6fe9" (UID: "13291a84-bbac-4efb-be09-e511060c6fe9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:00:42.293687 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:42.293652 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 31 20:00:42.293843 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:42.293718 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" podUID="55d3a124-18b2-4198-b3ee-7ae4180538f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 31 20:00:42.305134 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:42.305103 4910 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13291a84-bbac-4efb-be09-e511060c6fe9-metrics-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:42.305134 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:42.305133 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4s6j\" (UniqueName: \"kubernetes.io/projected/13291a84-bbac-4efb-be09-e511060c6fe9-kube-api-access-d4s6j\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:42.305317 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:42.305145 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13291a84-bbac-4efb-be09-e511060c6fe9-maas-api-tls\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:42.679334 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:42.679296 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" event={"ID":"13291a84-bbac-4efb-be09-e511060c6fe9","Type":"ContainerDied","Data":"fd112ecb79976800f8f2d013d72734e0332c71e9277e1ba8d9ccface4aff3571"} Oct 31 20:00:42.679822 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:42.679366 4910 scope.go:122] "RemoveContainer" containerID="b83655f027f4f20c35f02c3c8f588d3f4102896f214d281668cf4fcc6544738b" Oct 31 20:00:42.679822 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:42.679521 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47" Oct 31 20:00:42.715312 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:42.715270 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47"] Oct 31 20:00:42.737951 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:42.734935 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:00:42.758243 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:42.758206 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-01afb5-5584ff67f8-klk47"] Oct 31 20:00:43.292446 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:43.292405 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 31 20:00:43.292655 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:43.292472 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" podUID="55d3a124-18b2-4198-b3ee-7ae4180538f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 31 20:00:43.814318 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:43.814275 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13291a84-bbac-4efb-be09-e511060c6fe9" path="/var/lib/kubelet/pods/13291a84-bbac-4efb-be09-e511060c6fe9/volumes" Oct 31 20:00:44.294991 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:44.294112 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 31 20:00:44.294991 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:44.294204 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" podUID="55d3a124-18b2-4198-b3ee-7ae4180538f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 31 20:00:45.383445 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:45.383408 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:45.385174 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:45.385141 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:45.389544 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:45.389503 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:00:50.208680 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:50.208604 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l"] Oct 31 20:00:50.209410 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:50.208861 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" podUID="d75a25fa-2c40-4489-afda-aee9c5ca3610" containerName="istio-proxy" containerID="cri-o://bd15d662eefd3bda09440c08b264c658442767ff658fc00d528b09ab00d4e3b9" gracePeriod=30 Oct 31 20:00:50.977862 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:50.977819 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q"] Oct 31 20:00:50.978589 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:50.978572 4910 cpu_manager.go:414] "containerMap: removing container" podUID="cce6c21a-41e1-4e7e-ac6d-2cc90582803a" containerName="cleanup" Oct 31 20:00:50.978680 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:50.978591 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce6c21a-41e1-4e7e-ac6d-2cc90582803a" containerName="cleanup" Oct 31 20:00:50.978680 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:50.978625 4910 cpu_manager.go:414] "containerMap: removing container" podUID="13291a84-bbac-4efb-be09-e511060c6fe9" containerName="maas-api" Oct 31 20:00:50.978680 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:50.978631 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13291a84-bbac-4efb-be09-e511060c6fe9" containerName="maas-api" Oct 31 20:00:50.978827 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:50.978764 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="13291a84-bbac-4efb-be09-e511060c6fe9" containerName="maas-api" Oct 31 20:00:50.983047 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:50.983022 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q" Oct 31 20:00:50.989871 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:50.989847 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:00:50.990231 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:50.990210 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 20:00:50.995114 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:50.995092 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q"] Oct 31 20:00:51.120435 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:51.120395 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc2febc-bb66-461f-86d1-9144350e0f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q\" (UID: \"6fc2febc-bb66-461f-86d1-9144350e0f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q" Oct 31 20:00:51.120656 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:51.120490 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc2febc-bb66-461f-86d1-9144350e0f5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q\" (UID: \"6fc2febc-bb66-461f-86d1-9144350e0f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q" Oct 31 20:00:51.120656 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:51.120576 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc8c6\" (UniqueName: \"kubernetes.io/projected/6fc2febc-bb66-461f-86d1-9144350e0f5d-kube-api-access-cc8c6\") pod \"maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q\" (UID: \"6fc2febc-bb66-461f-86d1-9144350e0f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q" Oct 31 20:00:51.222320 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:51.222279 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc2febc-bb66-461f-86d1-9144350e0f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q\" (UID: \"6fc2febc-bb66-461f-86d1-9144350e0f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q" Oct 31 20:00:51.222823 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:51.222397 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc2febc-bb66-461f-86d1-9144350e0f5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q\" (UID: \"6fc2febc-bb66-461f-86d1-9144350e0f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q" Oct 31 20:00:51.222823 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:51.222503 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc8c6\" (UniqueName: \"kubernetes.io/projected/6fc2febc-bb66-461f-86d1-9144350e0f5d-kube-api-access-cc8c6\") pod \"maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q\" (UID: \"6fc2febc-bb66-461f-86d1-9144350e0f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q" Oct 31 20:00:51.223157 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:51.223135 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc2febc-bb66-461f-86d1-9144350e0f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q\" (UID: \"6fc2febc-bb66-461f-86d1-9144350e0f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q" Oct 31 20:00:51.225663 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:51.225634 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc2febc-bb66-461f-86d1-9144350e0f5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q\" (UID: \"6fc2febc-bb66-461f-86d1-9144350e0f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q" Oct 31 20:00:51.237519 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:51.237443 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc8c6\" (UniqueName: \"kubernetes.io/projected/6fc2febc-bb66-461f-86d1-9144350e0f5d-kube-api-access-cc8c6\") pod \"maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q\" (UID: \"6fc2febc-bb66-461f-86d1-9144350e0f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q" Oct 31 20:00:51.306305 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:51.306272 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q" Oct 31 20:00:51.628993 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:00:51.628885 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc2febc_bb66_461f_86d1_9144350e0f5d.slice/crio-105353b76e9149057f5362554c357ca058b70d19d92915b804b387f075ff5e24 WatchSource:0}: Error finding container 105353b76e9149057f5362554c357ca058b70d19d92915b804b387f075ff5e24: Status 404 returned error can't find the container with id 105353b76e9149057f5362554c357ca058b70d19d92915b804b387f075ff5e24 Oct 31 20:00:51.629809 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:51.629782 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q"] Oct 31 20:00:52.001425 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:52.001336 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q" event={"ID":"6fc2febc-bb66-461f-86d1-9144350e0f5d","Type":"ContainerStarted","Data":"105353b76e9149057f5362554c357ca058b70d19d92915b804b387f075ff5e24"} Oct 31 20:00:53.020589 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:53.020487 4910 generic.go:356] "Generic (PLEG): container finished" podID="6fc2febc-bb66-461f-86d1-9144350e0f5d" containerID="365eceb1dd10d827dec0ee7e95296d221746de55e034e06ac7cfa76d2612f3a0" exitCode=0 Oct 31 20:00:53.021010 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:53.020629 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q" event={"ID":"6fc2febc-bb66-461f-86d1-9144350e0f5d","Type":"ContainerDied","Data":"365eceb1dd10d827dec0ee7e95296d221746de55e034e06ac7cfa76d2612f3a0"} Oct 31 20:00:54.282300 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:54.282270 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q" Oct 31 20:00:54.394996 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:54.394881 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc2febc-bb66-461f-86d1-9144350e0f5d-kube-api-access-cc8c6\" (UniqueName: \"kubernetes.io/projected/6fc2febc-bb66-461f-86d1-9144350e0f5d-kube-api-access-cc8c6\") pod \"6fc2febc-bb66-461f-86d1-9144350e0f5d\" (UID: \"6fc2febc-bb66-461f-86d1-9144350e0f5d\") " Oct 31 20:00:54.394996 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:54.394943 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fc2febc-bb66-461f-86d1-9144350e0f5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc2febc-bb66-461f-86d1-9144350e0f5d-maas-api-openshift-ca\") pod \"6fc2febc-bb66-461f-86d1-9144350e0f5d\" (UID: \"6fc2febc-bb66-461f-86d1-9144350e0f5d\") " Oct 31 20:00:54.395412 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:54.395170 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fc2febc-bb66-461f-86d1-9144350e0f5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc2febc-bb66-461f-86d1-9144350e0f5d-maas-api-xks-ca\") pod \"6fc2febc-bb66-461f-86d1-9144350e0f5d\" (UID: \"6fc2febc-bb66-461f-86d1-9144350e0f5d\") " Oct 31 20:00:54.395412 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:54.395345 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fc2febc-bb66-461f-86d1-9144350e0f5d-maas-api-openshift-ca" pod "6fc2febc-bb66-461f-86d1-9144350e0f5d" (UID: "6fc2febc-bb66-461f-86d1-9144350e0f5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:54.395733 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:54.395705 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc2febc-bb66-461f-86d1-9144350e0f5d-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:54.397346 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:54.397321 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fc2febc-bb66-461f-86d1-9144350e0f5d-maas-api-xks-ca" pod "6fc2febc-bb66-461f-86d1-9144350e0f5d" (UID: "6fc2febc-bb66-461f-86d1-9144350e0f5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:00:54.397346 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:54.397337 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc2febc-bb66-461f-86d1-9144350e0f5d-kube-api-access-cc8c6" pod "6fc2febc-bb66-461f-86d1-9144350e0f5d" (UID: "6fc2febc-bb66-461f-86d1-9144350e0f5d"). InnerVolumeSpecName "kube-api-access-cc8c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:54.497022 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:54.496991 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc2febc-bb66-461f-86d1-9144350e0f5d-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:54.497022 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:54.497020 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc8c6\" (UniqueName: \"kubernetes.io/projected/6fc2febc-bb66-461f-86d1-9144350e0f5d-kube-api-access-cc8c6\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:55.106674 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.106638 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q" Oct 31 20:00:55.106674 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.106665 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q" event={"ID":"6fc2febc-bb66-461f-86d1-9144350e0f5d","Type":"ContainerDied","Data":"105353b76e9149057f5362554c357ca058b70d19d92915b804b387f075ff5e24"} Oct 31 20:00:55.106946 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.106724 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="105353b76e9149057f5362554c357ca058b70d19d92915b804b387f075ff5e24" Oct 31 20:00:55.636997 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.636532 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 20:00:55.747276 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.747244 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-workload-certs\") pod \"d75a25fa-2c40-4489-afda-aee9c5ca3610\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " Oct 31 20:00:55.747479 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.747339 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-podinfo\") pod \"d75a25fa-2c40-4489-afda-aee9c5ca3610\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " Oct 31 20:00:55.747479 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.747370 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d75a25fa-2c40-4489-afda-aee9c5ca3610-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75a25fa-2c40-4489-afda-aee9c5ca3610-istiod-ca-cert\") pod \"d75a25fa-2c40-4489-afda-aee9c5ca3610\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " Oct 31 20:00:55.747479 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.747445 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-credential-socket\") pod \"d75a25fa-2c40-4489-afda-aee9c5ca3610\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " Oct 31 20:00:55.747658 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.747489 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d75a25fa-2c40-4489-afda-aee9c5ca3610-kube-api-access-zm9xb\" (UniqueName: \"kubernetes.io/projected/d75a25fa-2c40-4489-afda-aee9c5ca3610-kube-api-access-zm9xb\") pod \"d75a25fa-2c40-4489-afda-aee9c5ca3610\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " Oct 31 20:00:55.747658 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.747537 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-workload-socket\") pod \"d75a25fa-2c40-4489-afda-aee9c5ca3610\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " Oct 31 20:00:55.747658 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.747536 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-workload-certs" pod "d75a25fa-2c40-4489-afda-aee9c5ca3610" (UID: "d75a25fa-2c40-4489-afda-aee9c5ca3610"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:00:55.747658 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.747580 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-token\" (UniqueName: \"kubernetes.io/projected/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-token\") pod \"d75a25fa-2c40-4489-afda-aee9c5ca3610\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " Oct 31 20:00:55.747873 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.747676 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-data\") pod \"d75a25fa-2c40-4489-afda-aee9c5ca3610\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " Oct 31 20:00:55.747873 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.747743 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-credential-socket" pod "d75a25fa-2c40-4489-afda-aee9c5ca3610" (UID: "d75a25fa-2c40-4489-afda-aee9c5ca3610"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:00:55.747873 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.747851 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d75a25fa-2c40-4489-afda-aee9c5ca3610-istiod-ca-cert" pod "d75a25fa-2c40-4489-afda-aee9c5ca3610" (UID: "d75a25fa-2c40-4489-afda-aee9c5ca3610"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:00:55.748203 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.747997 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-workload-socket" pod "d75a25fa-2c40-4489-afda-aee9c5ca3610" (UID: "d75a25fa-2c40-4489-afda-aee9c5ca3610"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:00:55.748203 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.747860 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-envoy\") pod \"d75a25fa-2c40-4489-afda-aee9c5ca3610\" (UID: \"d75a25fa-2c40-4489-afda-aee9c5ca3610\") " Oct 31 20:00:55.748523 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.748488 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-data" pod "d75a25fa-2c40-4489-afda-aee9c5ca3610" (UID: "d75a25fa-2c40-4489-afda-aee9c5ca3610"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:00:55.748983 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.748955 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:55.749087 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.748988 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75a25fa-2c40-4489-afda-aee9c5ca3610-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:55.749087 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.749018 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:55.749087 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.749033 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:55.749087 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.749046 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:55.750347 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.750301 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75a25fa-2c40-4489-afda-aee9c5ca3610-kube-api-access-zm9xb" pod "d75a25fa-2c40-4489-afda-aee9c5ca3610" (UID: "d75a25fa-2c40-4489-afda-aee9c5ca3610"). InnerVolumeSpecName "kube-api-access-zm9xb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:55.750874 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.750847 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-podinfo" pod "d75a25fa-2c40-4489-afda-aee9c5ca3610" (UID: "d75a25fa-2c40-4489-afda-aee9c5ca3610"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:00:55.751358 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.751307 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-envoy" pod "d75a25fa-2c40-4489-afda-aee9c5ca3610" (UID: "d75a25fa-2c40-4489-afda-aee9c5ca3610"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:00:55.752226 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.752189 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-token" pod "d75a25fa-2c40-4489-afda-aee9c5ca3610" (UID: "d75a25fa-2c40-4489-afda-aee9c5ca3610"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:00:55.851783 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.851745 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:55.851783 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.851783 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm9xb\" (UniqueName: \"kubernetes.io/projected/d75a25fa-2c40-4489-afda-aee9c5ca3610-kube-api-access-zm9xb\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:55.851783 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.851802 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:55.852127 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:55.851816 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75a25fa-2c40-4489-afda-aee9c5ca3610-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:00:56.117132 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:56.117093 4910 generic.go:356] "Generic (PLEG): container finished" podID="d75a25fa-2c40-4489-afda-aee9c5ca3610" containerID="bd15d662eefd3bda09440c08b264c658442767ff658fc00d528b09ab00d4e3b9" exitCode=0 Oct 31 20:00:56.117346 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:56.117161 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" Oct 31 20:00:56.117346 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:56.117175 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" event={"ID":"d75a25fa-2c40-4489-afda-aee9c5ca3610","Type":"ContainerDied","Data":"bd15d662eefd3bda09440c08b264c658442767ff658fc00d528b09ab00d4e3b9"} Oct 31 20:00:56.117346 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:56.117248 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l" event={"ID":"d75a25fa-2c40-4489-afda-aee9c5ca3610","Type":"ContainerDied","Data":"de4562e3982352fdd27d284620ecd11e11822e222d24a7173da67b56f045e0c5"} Oct 31 20:00:56.117346 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:56.117281 4910 scope.go:122] "RemoveContainer" containerID="bd15d662eefd3bda09440c08b264c658442767ff658fc00d528b09ab00d4e3b9" Oct 31 20:00:56.129586 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:56.129568 4910 scope.go:122] "RemoveContainer" containerID="bd15d662eefd3bda09440c08b264c658442767ff658fc00d528b09ab00d4e3b9" Oct 31 20:00:56.129957 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:00:56.129896 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd15d662eefd3bda09440c08b264c658442767ff658fc00d528b09ab00d4e3b9\": container with ID starting with bd15d662eefd3bda09440c08b264c658442767ff658fc00d528b09ab00d4e3b9 not found: ID does not exist" containerID="bd15d662eefd3bda09440c08b264c658442767ff658fc00d528b09ab00d4e3b9" Oct 31 20:00:56.130067 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:56.129951 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd15d662eefd3bda09440c08b264c658442767ff658fc00d528b09ab00d4e3b9"} err="failed to get container status \"bd15d662eefd3bda09440c08b264c658442767ff658fc00d528b09ab00d4e3b9\": rpc error: code = NotFound desc = could not find container \"bd15d662eefd3bda09440c08b264c658442767ff658fc00d528b09ab00d4e3b9\": container with ID starting with bd15d662eefd3bda09440c08b264c658442767ff658fc00d528b09ab00d4e3b9 not found: ID does not exist" Oct 31 20:00:56.143048 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:56.143021 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l"] Oct 31 20:00:56.149981 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:56.149956 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-gtz5l"] Oct 31 20:00:57.815462 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:00:57.815427 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d75a25fa-2c40-4489-afda-aee9c5ca3610" path="/var/lib/kubelet/pods/d75a25fa-2c40-4489-afda-aee9c5ca3610/volumes" Oct 31 20:01:04.503397 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:04.503359 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm"] Oct 31 20:01:04.504440 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:04.504420 4910 cpu_manager.go:414] "containerMap: removing container" podUID="6fc2febc-bb66-461f-86d1-9144350e0f5d" containerName="revoke-keys" Oct 31 20:01:04.504440 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:04.504440 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc2febc-bb66-461f-86d1-9144350e0f5d" containerName="revoke-keys" Oct 31 20:01:04.504601 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:04.504478 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d75a25fa-2c40-4489-afda-aee9c5ca3610" containerName="istio-proxy" Oct 31 20:01:04.504601 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:04.504486 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d75a25fa-2c40-4489-afda-aee9c5ca3610" containerName="istio-proxy" Oct 31 20:01:04.504674 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:04.504631 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc2febc-bb66-461f-86d1-9144350e0f5d" containerName="revoke-keys" Oct 31 20:01:04.504674 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:04.504649 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d75a25fa-2c40-4489-afda-aee9c5ca3610" containerName="istio-proxy" Oct 31 20:01:04.511052 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:04.511027 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" Oct 31 20:01:04.529007 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:04.528127 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-bd1c9f-serving-cert\"" Oct 31 20:01:04.531871 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:04.531840 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm"] Oct 31 20:01:04.565958 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:04.565123 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15884cef-ec21-4fe7-a8b7-f04cf48f4637-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm\" (UID: \"15884cef-ec21-4fe7-a8b7-f04cf48f4637\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" Oct 31 20:01:04.565958 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:04.565386 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15884cef-ec21-4fe7-a8b7-f04cf48f4637-metrics-certs\") pod \"maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm\" (UID: \"15884cef-ec21-4fe7-a8b7-f04cf48f4637\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" Oct 31 20:01:04.565958 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:04.565421 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lk58\" (UniqueName: \"kubernetes.io/projected/15884cef-ec21-4fe7-a8b7-f04cf48f4637-kube-api-access-5lk58\") pod \"maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm\" (UID: \"15884cef-ec21-4fe7-a8b7-f04cf48f4637\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" Oct 31 20:01:04.565958 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:04.565519 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15884cef-ec21-4fe7-a8b7-f04cf48f4637-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm\" (UID: \"15884cef-ec21-4fe7-a8b7-f04cf48f4637\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" Oct 31 20:01:04.699768 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:04.694949 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15884cef-ec21-4fe7-a8b7-f04cf48f4637-metrics-certs\") pod \"maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm\" (UID: \"15884cef-ec21-4fe7-a8b7-f04cf48f4637\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" Oct 31 20:01:04.699768 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:04.695015 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lk58\" (UniqueName: \"kubernetes.io/projected/15884cef-ec21-4fe7-a8b7-f04cf48f4637-kube-api-access-5lk58\") pod \"maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm\" (UID: \"15884cef-ec21-4fe7-a8b7-f04cf48f4637\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" Oct 31 20:01:04.699768 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:04.695169 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15884cef-ec21-4fe7-a8b7-f04cf48f4637-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm\" (UID: \"15884cef-ec21-4fe7-a8b7-f04cf48f4637\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" Oct 31 20:01:04.699768 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:04.695443 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15884cef-ec21-4fe7-a8b7-f04cf48f4637-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm\" (UID: \"15884cef-ec21-4fe7-a8b7-f04cf48f4637\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" Oct 31 20:01:04.699768 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:01:04.695877 4910 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-serving-cert: secret "maas-api-e2e-ipp-a-bd1c9f-serving-cert" not found Oct 31 20:01:04.699768 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:01:04.696005 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15884cef-ec21-4fe7-a8b7-f04cf48f4637-maas-api-tls podName:15884cef-ec21-4fe7-a8b7-f04cf48f4637 nodeName:}" failed. No retries permitted until 2026-10-31 20:01:05.19598016 +0000 UTC m=+2499.558800511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/15884cef-ec21-4fe7-a8b7-f04cf48f4637-maas-api-tls") pod "maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" (UID: "15884cef-ec21-4fe7-a8b7-f04cf48f4637") : secret "maas-api-e2e-ipp-a-bd1c9f-serving-cert" not found Oct 31 20:01:04.712681 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:04.712637 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15884cef-ec21-4fe7-a8b7-f04cf48f4637-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm\" (UID: \"15884cef-ec21-4fe7-a8b7-f04cf48f4637\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" Oct 31 20:01:04.720931 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:04.716996 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15884cef-ec21-4fe7-a8b7-f04cf48f4637-metrics-certs\") pod \"maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm\" (UID: \"15884cef-ec21-4fe7-a8b7-f04cf48f4637\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" Oct 31 20:01:04.744953 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:04.738562 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lk58\" (UniqueName: \"kubernetes.io/projected/15884cef-ec21-4fe7-a8b7-f04cf48f4637-kube-api-access-5lk58\") pod \"maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm\" (UID: \"15884cef-ec21-4fe7-a8b7-f04cf48f4637\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" Oct 31 20:01:05.205264 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:05.205218 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15884cef-ec21-4fe7-a8b7-f04cf48f4637-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm\" (UID: \"15884cef-ec21-4fe7-a8b7-f04cf48f4637\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" Oct 31 20:01:05.208011 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:05.207980 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15884cef-ec21-4fe7-a8b7-f04cf48f4637-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm\" (UID: \"15884cef-ec21-4fe7-a8b7-f04cf48f4637\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" Oct 31 20:01:05.442241 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:05.442199 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" Oct 31 20:01:05.865275 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:01:05.865238 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15884cef_ec21_4fe7_a8b7_f04cf48f4637.slice/crio-2d096ee4d1dc42af1f37e0f43695a3af82132fe8c530a2e0717f02589cefbc47 WatchSource:0}: Error finding container 2d096ee4d1dc42af1f37e0f43695a3af82132fe8c530a2e0717f02589cefbc47: Status 404 returned error can't find the container with id 2d096ee4d1dc42af1f37e0f43695a3af82132fe8c530a2e0717f02589cefbc47 Oct 31 20:01:05.870059 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:05.870028 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm"] Oct 31 20:01:06.281892 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:06.281811 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" event={"ID":"15884cef-ec21-4fe7-a8b7-f04cf48f4637","Type":"ContainerStarted","Data":"2d096ee4d1dc42af1f37e0f43695a3af82132fe8c530a2e0717f02589cefbc47"} Oct 31 20:01:07.376241 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:07.376192 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" event={"ID":"15884cef-ec21-4fe7-a8b7-f04cf48f4637","Type":"ContainerStarted","Data":"dae9ab14c7b29b0ece59205b847ced4e4b4ff1f1115cb998fc6affd2f0f97b55"} Oct 31 20:01:07.377673 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:07.377647 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" Oct 31 20:01:07.405807 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:07.405750 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" podStartSLOduration=2.982251036 podStartE2EDuration="3.405737921s" totalImagesPullingTime="423.486885ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:01:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:01:07.401622275 +0000 UTC m=+2501.764442648" watchObservedRunningTime="2026-10-31 20:01:07.405737921 +0000 UTC m=+2501.768558332" Oct 31 20:01:12.502098 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:12.501963 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9"] Oct 31 20:01:12.509934 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:12.508156 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9" Oct 31 20:01:12.541004 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:12.534307 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 20:01:12.541004 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:12.534360 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:01:12.544599 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:12.544563 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9"] Oct 31 20:01:12.590642 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:12.590601 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjb5s\" (UniqueName: \"kubernetes.io/projected/bbd3e5b5-b652-4f8f-9666-2508637b9697-kube-api-access-kjb5s\") pod \"maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9\" (UID: \"bbd3e5b5-b652-4f8f-9666-2508637b9697\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9" Oct 31 20:01:12.590839 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:12.590760 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd3e5b5-b652-4f8f-9666-2508637b9697-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9\" (UID: \"bbd3e5b5-b652-4f8f-9666-2508637b9697\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9" Oct 31 20:01:12.590982 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:12.590966 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd3e5b5-b652-4f8f-9666-2508637b9697-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9\" (UID: \"bbd3e5b5-b652-4f8f-9666-2508637b9697\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9" Oct 31 20:01:12.692161 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:12.692125 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd3e5b5-b652-4f8f-9666-2508637b9697-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9\" (UID: \"bbd3e5b5-b652-4f8f-9666-2508637b9697\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9" Oct 31 20:01:12.692349 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:12.692276 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjb5s\" (UniqueName: \"kubernetes.io/projected/bbd3e5b5-b652-4f8f-9666-2508637b9697-kube-api-access-kjb5s\") pod \"maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9\" (UID: \"bbd3e5b5-b652-4f8f-9666-2508637b9697\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9" Oct 31 20:01:12.692349 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:12.692340 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd3e5b5-b652-4f8f-9666-2508637b9697-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9\" (UID: \"bbd3e5b5-b652-4f8f-9666-2508637b9697\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9" Oct 31 20:01:12.693022 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:12.692992 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd3e5b5-b652-4f8f-9666-2508637b9697-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9\" (UID: \"bbd3e5b5-b652-4f8f-9666-2508637b9697\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9" Oct 31 20:01:12.694723 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:12.694700 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd3e5b5-b652-4f8f-9666-2508637b9697-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9\" (UID: \"bbd3e5b5-b652-4f8f-9666-2508637b9697\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9" Oct 31 20:01:12.705381 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:12.705357 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjb5s\" (UniqueName: \"kubernetes.io/projected/bbd3e5b5-b652-4f8f-9666-2508637b9697-kube-api-access-kjb5s\") pod \"maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9\" (UID: \"bbd3e5b5-b652-4f8f-9666-2508637b9697\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9" Oct 31 20:01:12.716199 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:01:12.716155 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:01:12.868281 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:12.868251 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9" Oct 31 20:01:13.155285 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:01:13.155246 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd3e5b5_b652_4f8f_9666_2508637b9697.slice/crio-1388a302e6cfdfc5d11d1230264c3e20d5ae2e162c86c19339b078d34447dc5f WatchSource:0}: Error finding container 1388a302e6cfdfc5d11d1230264c3e20d5ae2e162c86c19339b078d34447dc5f: Status 404 returned error can't find the container with id 1388a302e6cfdfc5d11d1230264c3e20d5ae2e162c86c19339b078d34447dc5f Oct 31 20:01:13.158820 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:13.158791 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9"] Oct 31 20:01:13.455119 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:13.455028 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9" event={"ID":"bbd3e5b5-b652-4f8f-9666-2508637b9697","Type":"ContainerStarted","Data":"1388a302e6cfdfc5d11d1230264c3e20d5ae2e162c86c19339b078d34447dc5f"} Oct 31 20:01:14.399640 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:14.399606 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" Oct 31 20:01:14.468452 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:14.468389 4910 generic.go:356] "Generic (PLEG): container finished" podID="bbd3e5b5-b652-4f8f-9666-2508637b9697" containerID="d5ccb6440d13962054e8f787e4ce51dd9c5f794ce74f89bdfd581c19d6dda0f6" exitCode=0 Oct 31 20:01:14.468736 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:14.468690 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9" event={"ID":"bbd3e5b5-b652-4f8f-9666-2508637b9697","Type":"ContainerDied","Data":"d5ccb6440d13962054e8f787e4ce51dd9c5f794ce74f89bdfd581c19d6dda0f6"} Oct 31 20:01:16.506936 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:16.504477 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Oct 31 20:01:16.506936 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:16.505037 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0639dd8-6pvtb"] Oct 31 20:01:16.597695 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:16.596444 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 31 20:01:16.597695 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:16.596754 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 20:01:16.597695 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:16.596789 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0639dd8-6pvtb" Oct 31 20:01:16.652986 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:16.599173 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" podUID="c20b518f-fe19-47b6-88a6-aec10c91cf71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.244:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:01:16.701392 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:16.688373 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" podUID="8fa2f1d6-abaa-4c73-8fdc-b91691bfb030" containerName="main" probeResult="failure" output="Get \"https://10.132.0.243:8000/ready\": context deadline exceeded" Oct 31 20:01:16.731497 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:16.703788 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:01:16.981392 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:16.732893 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": context deadline exceeded" Oct 31 20:01:16.981692 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:16.784630 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 20:01:16.982387 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:16.796193 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47sv9\" (UniqueName: \"kubernetes.io/projected/627b9644-f20c-4ae1-b906-f4d8f35ee8b7-kube-api-access-47sv9\") pod \"maas-api-revoke-sub-e0639dd8-6pvtb\" (UID: \"627b9644-f20c-4ae1-b906-f4d8f35ee8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0639dd8-6pvtb" Oct 31 20:01:16.982921 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:16.982728 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627b9644-f20c-4ae1-b906-f4d8f35ee8b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0639dd8-6pvtb\" (UID: \"627b9644-f20c-4ae1-b906-f4d8f35ee8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0639dd8-6pvtb" Oct 31 20:01:17.015417 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:16.886211 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 20:01:17.017605 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:16.913658 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 20:01:17.017605 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:16.798090 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:01:17.028953 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:17.018975 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627b9644-f20c-4ae1-b906-f4d8f35ee8b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0639dd8-6pvtb\" (UID: \"627b9644-f20c-4ae1-b906-f4d8f35ee8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0639dd8-6pvtb" Oct 31 20:01:17.112008 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:17.095281 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:01:17.167940 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:17.142565 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 20:01:17.296971 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:17.201062 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:01:17.296971 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:17.203719 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627b9644-f20c-4ae1-b906-f4d8f35ee8b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0639dd8-6pvtb\" (UID: \"627b9644-f20c-4ae1-b906-f4d8f35ee8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0639dd8-6pvtb" Oct 31 20:01:17.296971 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:17.251552 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627b9644-f20c-4ae1-b906-f4d8f35ee8b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0639dd8-6pvtb\" (UID: \"627b9644-f20c-4ae1-b906-f4d8f35ee8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0639dd8-6pvtb" Oct 31 20:01:17.296971 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:17.252121 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47sv9\" (UniqueName: \"kubernetes.io/projected/627b9644-f20c-4ae1-b906-f4d8f35ee8b7-kube-api-access-47sv9\") pod \"maas-api-revoke-sub-e0639dd8-6pvtb\" (UID: \"627b9644-f20c-4ae1-b906-f4d8f35ee8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0639dd8-6pvtb" Oct 31 20:01:17.296971 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:17.253376 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627b9644-f20c-4ae1-b906-f4d8f35ee8b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0639dd8-6pvtb\" (UID: \"627b9644-f20c-4ae1-b906-f4d8f35ee8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0639dd8-6pvtb" Oct 31 20:01:17.327279 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:17.327230 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:01:17.389982 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:17.378479 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627b9644-f20c-4ae1-b906-f4d8f35ee8b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0639dd8-6pvtb\" (UID: \"627b9644-f20c-4ae1-b906-f4d8f35ee8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0639dd8-6pvtb" Oct 31 20:01:17.432946 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:16.597903 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 20:01:17.535995 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:17.516311 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:01:17.535995 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:17.520766 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 20:01:17.569967 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:17.549893 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47sv9\" (UniqueName: \"kubernetes.io/projected/627b9644-f20c-4ae1-b906-f4d8f35ee8b7-kube-api-access-47sv9\") pod \"maas-api-revoke-sub-e0639dd8-6pvtb\" (UID: \"627b9644-f20c-4ae1-b906-f4d8f35ee8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0639dd8-6pvtb" Oct 31 20:01:17.629950 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:17.629443 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:01:17.657132 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:17.652063 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0639dd8-6pvtb"] Oct 31 20:01:17.657132 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:17.652238 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 20:01:17.704953 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:17.692283 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" podUID="c20b518f-fe19-47b6-88a6-aec10c91cf71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.244:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:01:17.704953 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:17.693144 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 20:01:17.742395 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:17.724613 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" podUID="8fa2f1d6-abaa-4c73-8fdc-b91691bfb030" containerName="main" probeResult="failure" output="Get \"https://10.132.0.243:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:01:17.770126 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:17.762567 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 20:01:18.153700 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.124390 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0639dd8-6pvtb" Oct 31 20:01:18.216372 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.216303 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:01:18.217125 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.217088 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:01:18.223564 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.219084 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded" Oct 31 20:01:18.237011 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.219151 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 31 20:01:18.237567 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.219177 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 20:01:18.457866 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.457747 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 31 20:01:18.518712 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.518617 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:01:18.520526 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.520487 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": context deadline exceeded" Oct 31 20:01:18.667932 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.667862 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:01:18.692965 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.680388 4910 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" podUID="8fa2f1d6-abaa-4c73-8fdc-b91691bfb030" containerName="main" probeResult="failure" output="Get \"https://10.132.0.243:8000/health\": context deadline exceeded" Oct 31 20:01:18.692965 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.684075 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:01:18.753322 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.734747 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 31 20:01:18.753322 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.735593 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 20:01:18.753322 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.752997 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" podUID="8fa2f1d6-abaa-4c73-8fdc-b91691bfb030" containerName="main" probeResult="failure" output="Get \"https://10.132.0.243:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:01:18.768840 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.754459 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded" Oct 31 20:01:18.769203 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.769167 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" podUID="c20b518f-fe19-47b6-88a6-aec10c91cf71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.244:8000/ready\": context deadline exceeded" Oct 31 20:01:18.898667 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.772807 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded" Oct 31 20:01:18.915100 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.775079 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9" Oct 31 20:01:18.915100 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:01:18.845297 4910 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.075s" Oct 31 20:01:18.915100 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.909824 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 20:01:18.915100 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.909871 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 20:01:18.915100 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.909929 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 20:01:18.915100 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.909968 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 20:01:18.915100 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.910007 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 20:01:18.915100 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.910046 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 20:01:18.915100 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.910088 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 20:01:18.915100 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.910138 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 20:01:18.915100 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:18.910174 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 20:01:19.122669 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.122528 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd3e5b5-b652-4f8f-9666-2508637b9697-kube-api-access-kjb5s\" (UniqueName: \"kubernetes.io/projected/bbd3e5b5-b652-4f8f-9666-2508637b9697-kube-api-access-kjb5s\") pod \"bbd3e5b5-b652-4f8f-9666-2508637b9697\" (UID: \"bbd3e5b5-b652-4f8f-9666-2508637b9697\") " Oct 31 20:01:19.123703 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.123667 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbd3e5b5-b652-4f8f-9666-2508637b9697-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd3e5b5-b652-4f8f-9666-2508637b9697-maas-api-xks-ca\") pod \"bbd3e5b5-b652-4f8f-9666-2508637b9697\" (UID: \"bbd3e5b5-b652-4f8f-9666-2508637b9697\") " Oct 31 20:01:19.123901 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.123879 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbd3e5b5-b652-4f8f-9666-2508637b9697-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd3e5b5-b652-4f8f-9666-2508637b9697-maas-api-openshift-ca\") pod \"bbd3e5b5-b652-4f8f-9666-2508637b9697\" (UID: \"bbd3e5b5-b652-4f8f-9666-2508637b9697\") " Oct 31 20:01:19.147297 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.135541 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd3e5b5-b652-4f8f-9666-2508637b9697-maas-api-openshift-ca" pod "bbd3e5b5-b652-4f8f-9666-2508637b9697" (UID: "bbd3e5b5-b652-4f8f-9666-2508637b9697"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:01:19.256948 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.256894 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 20:01:19.265710 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.265674 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd3e5b5-b652-4f8f-9666-2508637b9697-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:19.278554 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.274389 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:01:19.279236 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.279195 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 31 20:01:19.280789 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.280762 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 31 20:01:19.308750 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.308702 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:01:19.381387 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.360089 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:01:19.381387 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.369178 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbd3e5b5-b652-4f8f-9666-2508637b9697-maas-api-xks-ca" pod "bbd3e5b5-b652-4f8f-9666-2508637b9697" (UID: "bbd3e5b5-b652-4f8f-9666-2508637b9697"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:01:19.381387 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.377933 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9" event={"ID":"bbd3e5b5-b652-4f8f-9666-2508637b9697","Type":"ContainerDied","Data":"1388a302e6cfdfc5d11d1230264c3e20d5ae2e162c86c19339b078d34447dc5f"} Oct 31 20:01:19.381387 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.377993 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1388a302e6cfdfc5d11d1230264c3e20d5ae2e162c86c19339b078d34447dc5f" Oct 31 20:01:19.381387 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.378068 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9" Oct 31 20:01:19.384642 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.384616 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 20:01:19.412353 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.411292 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 20:01:19.412353 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.412308 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd3e5b5-b652-4f8f-9666-2508637b9697-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:19.429685 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.429651 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 20:01:19.444858 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.441462 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 20:01:19.521930 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.521059 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 20:01:19.553003 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.551644 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd3e5b5-b652-4f8f-9666-2508637b9697-kube-api-access-kjb5s" pod "bbd3e5b5-b652-4f8f-9666-2508637b9697" (UID: "bbd3e5b5-b652-4f8f-9666-2508637b9697"). InnerVolumeSpecName "kube-api-access-kjb5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:01:19.578226 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.578174 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 20:01:19.597634 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.597603 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 20:01:19.610355 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.610322 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 20:01:19.615457 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.610786 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 20:01:19.647524 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.646570 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 20:01:19.664121 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.664082 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjb5s\" (UniqueName: \"kubernetes.io/projected/bbd3e5b5-b652-4f8f-9666-2508637b9697-kube-api-access-kjb5s\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:19.681265 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.681241 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 20:01:19.682311 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.682288 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 20:01:19.906184 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:01:19.906148 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod627b9644_f20c_4ae1_b906_f4d8f35ee8b7.slice/crio-05b0bd57328f2270bdf527e3ab782225935d82dcc89dff6ae4c02d0643db95d7 WatchSource:0}: Error finding container 05b0bd57328f2270bdf527e3ab782225935d82dcc89dff6ae4c02d0643db95d7: Status 404 returned error can't find the container with id 05b0bd57328f2270bdf527e3ab782225935d82dcc89dff6ae4c02d0643db95d7 Oct 31 20:01:19.908893 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:19.908871 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0639dd8-6pvtb"] Oct 31 20:01:20.392859 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:20.392819 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0639dd8-6pvtb" event={"ID":"627b9644-f20c-4ae1-b906-f4d8f35ee8b7","Type":"ContainerStarted","Data":"05b0bd57328f2270bdf527e3ab782225935d82dcc89dff6ae4c02d0643db95d7"} Oct 31 20:01:21.485936 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:21.472818 4910 generic.go:356] "Generic (PLEG): container finished" podID="627b9644-f20c-4ae1-b906-f4d8f35ee8b7" containerID="f20adc98ae4a1f1417ecebbb0124a14ae603c17cbc51097b0ab8e4fe030696b2" exitCode=0 Oct 31 20:01:21.485936 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:21.472874 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0639dd8-6pvtb" event={"ID":"627b9644-f20c-4ae1-b906-f4d8f35ee8b7","Type":"ContainerDied","Data":"f20adc98ae4a1f1417ecebbb0124a14ae603c17cbc51097b0ab8e4fe030696b2"} Oct 31 20:01:22.739165 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:22.739136 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0639dd8-6pvtb" Oct 31 20:01:22.755819 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:22.755785 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/627b9644-f20c-4ae1-b906-f4d8f35ee8b7-kube-api-access-47sv9\" (UniqueName: \"kubernetes.io/projected/627b9644-f20c-4ae1-b906-f4d8f35ee8b7-kube-api-access-47sv9\") pod \"627b9644-f20c-4ae1-b906-f4d8f35ee8b7\" (UID: \"627b9644-f20c-4ae1-b906-f4d8f35ee8b7\") " Oct 31 20:01:22.755999 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:22.755980 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/627b9644-f20c-4ae1-b906-f4d8f35ee8b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627b9644-f20c-4ae1-b906-f4d8f35ee8b7-maas-api-openshift-ca\") pod \"627b9644-f20c-4ae1-b906-f4d8f35ee8b7\" (UID: \"627b9644-f20c-4ae1-b906-f4d8f35ee8b7\") " Oct 31 20:01:22.756088 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:22.756072 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/627b9644-f20c-4ae1-b906-f4d8f35ee8b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627b9644-f20c-4ae1-b906-f4d8f35ee8b7-maas-api-xks-ca\") pod \"627b9644-f20c-4ae1-b906-f4d8f35ee8b7\" (UID: \"627b9644-f20c-4ae1-b906-f4d8f35ee8b7\") " Oct 31 20:01:22.756418 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:22.756391 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/627b9644-f20c-4ae1-b906-f4d8f35ee8b7-maas-api-openshift-ca" pod "627b9644-f20c-4ae1-b906-f4d8f35ee8b7" (UID: "627b9644-f20c-4ae1-b906-f4d8f35ee8b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:01:22.756889 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:22.756869 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/627b9644-f20c-4ae1-b906-f4d8f35ee8b7-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:22.758266 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:22.758233 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/627b9644-f20c-4ae1-b906-f4d8f35ee8b7-kube-api-access-47sv9" pod "627b9644-f20c-4ae1-b906-f4d8f35ee8b7" (UID: "627b9644-f20c-4ae1-b906-f4d8f35ee8b7"). InnerVolumeSpecName "kube-api-access-47sv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:01:22.758360 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:22.758337 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/627b9644-f20c-4ae1-b906-f4d8f35ee8b7-maas-api-xks-ca" pod "627b9644-f20c-4ae1-b906-f4d8f35ee8b7" (UID: "627b9644-f20c-4ae1-b906-f4d8f35ee8b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:01:22.858405 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:22.858372 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47sv9\" (UniqueName: \"kubernetes.io/projected/627b9644-f20c-4ae1-b906-f4d8f35ee8b7-kube-api-access-47sv9\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:22.858405 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:22.858403 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/627b9644-f20c-4ae1-b906-f4d8f35ee8b7-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:23.570424 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:23.568761 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0639dd8-6pvtb" event={"ID":"627b9644-f20c-4ae1-b906-f4d8f35ee8b7","Type":"ContainerDied","Data":"05b0bd57328f2270bdf527e3ab782225935d82dcc89dff6ae4c02d0643db95d7"} Oct 31 20:01:23.570424 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:23.568825 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05b0bd57328f2270bdf527e3ab782225935d82dcc89dff6ae4c02d0643db95d7" Oct 31 20:01:23.570424 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:23.568891 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0639dd8-6pvtb" Oct 31 20:01:24.467461 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:24.467425 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h"] Oct 31 20:01:24.467965 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:24.467681 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" podUID="89529dc6-ab2a-4b03-a4a9-56f43c12fc31" containerName="istio-proxy" containerID="cri-o://36932c944a0bf80b48e898f2ef30d4cbb8efa678ea14ea96a20efe826b3c82a1" gracePeriod=30 Oct 31 20:01:24.916107 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:24.916066 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9"] Oct 31 20:01:25.053935 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:25.035122 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-26cd59-68fd9"] Oct 31 20:01:25.761853 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:25.761815 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbd3e5b5-b652-4f8f-9666-2508637b9697" path="/var/lib/kubelet/pods/bbd3e5b5-b652-4f8f-9666-2508637b9697/volumes" Oct 31 20:01:29.886935 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:29.857451 4910 generic.go:356] "Generic (PLEG): container finished" podID="89529dc6-ab2a-4b03-a4a9-56f43c12fc31" containerID="36932c944a0bf80b48e898f2ef30d4cbb8efa678ea14ea96a20efe826b3c82a1" exitCode=0 Oct 31 20:01:29.886935 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:29.878126 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" event={"ID":"89529dc6-ab2a-4b03-a4a9-56f43c12fc31","Type":"ContainerDied","Data":"36932c944a0bf80b48e898f2ef30d4cbb8efa678ea14ea96a20efe826b3c82a1"} Oct 31 20:01:30.059791 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.059751 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 20:01:30.098023 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.097988 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-data\") pod \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " Oct 31 20:01:30.098225 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.098065 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-podinfo\") pod \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " Oct 31 20:01:30.098225 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.098159 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-workload-certs\") pod \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " Oct 31 20:01:30.098225 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.098206 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-credential-socket\") pod \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " Oct 31 20:01:30.098404 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.098263 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-token\" (UniqueName: \"kubernetes.io/projected/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-token\") pod \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " Oct 31 20:01:30.098404 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.098290 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-data" pod "89529dc6-ab2a-4b03-a4a9-56f43c12fc31" (UID: "89529dc6-ab2a-4b03-a4a9-56f43c12fc31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:01:30.098404 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.098330 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-workload-socket\") pod \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " Oct 31 20:01:30.098564 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.098486 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-workload-certs" pod "89529dc6-ab2a-4b03-a4a9-56f43c12fc31" (UID: "89529dc6-ab2a-4b03-a4a9-56f43c12fc31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:01:30.098564 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.098502 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istiod-ca-cert\") pod \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " Oct 31 20:01:30.098564 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.098506 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-credential-socket" pod "89529dc6-ab2a-4b03-a4a9-56f43c12fc31" (UID: "89529dc6-ab2a-4b03-a4a9-56f43c12fc31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:01:30.098723 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.098584 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-envoy\") pod \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " Oct 31 20:01:30.098723 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.098650 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-workload-socket" pod "89529dc6-ab2a-4b03-a4a9-56f43c12fc31" (UID: "89529dc6-ab2a-4b03-a4a9-56f43c12fc31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:01:30.098835 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.098727 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-kube-api-access-fvnxv\" (UniqueName: \"kubernetes.io/projected/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-kube-api-access-fvnxv\") pod \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\" (UID: \"89529dc6-ab2a-4b03-a4a9-56f43c12fc31\") " Oct 31 20:01:30.098961 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.098898 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istiod-ca-cert" pod "89529dc6-ab2a-4b03-a4a9-56f43c12fc31" (UID: "89529dc6-ab2a-4b03-a4a9-56f43c12fc31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:01:30.099658 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.099628 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:30.099789 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.099661 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:30.099789 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.099678 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:30.099789 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.099691 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:30.099789 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.099706 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:30.100548 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.100524 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-podinfo" pod "89529dc6-ab2a-4b03-a4a9-56f43c12fc31" (UID: "89529dc6-ab2a-4b03-a4a9-56f43c12fc31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:01:30.100856 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.100831 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-token" pod "89529dc6-ab2a-4b03-a4a9-56f43c12fc31" (UID: "89529dc6-ab2a-4b03-a4a9-56f43c12fc31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:01:30.101017 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.100877 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-envoy" pod "89529dc6-ab2a-4b03-a4a9-56f43c12fc31" (UID: "89529dc6-ab2a-4b03-a4a9-56f43c12fc31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:01:30.101464 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.101439 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-kube-api-access-fvnxv" pod "89529dc6-ab2a-4b03-a4a9-56f43c12fc31" (UID: "89529dc6-ab2a-4b03-a4a9-56f43c12fc31"). InnerVolumeSpecName "kube-api-access-fvnxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:01:30.209051 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.208960 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvnxv\" (UniqueName: \"kubernetes.io/projected/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-kube-api-access-fvnxv\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:30.209051 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.208995 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:30.209051 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.209012 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:30.209051 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.209023 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89529dc6-ab2a-4b03-a4a9-56f43c12fc31-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:30.869822 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.869785 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" Oct 31 20:01:30.870045 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.869816 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h" event={"ID":"89529dc6-ab2a-4b03-a4a9-56f43c12fc31","Type":"ContainerDied","Data":"9577db6fb80ab53931d13b4515dcec571083607e518a9985774492c1fe163ef4"} Oct 31 20:01:30.870045 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.869876 4910 scope.go:122] "RemoveContainer" containerID="36932c944a0bf80b48e898f2ef30d4cbb8efa678ea14ea96a20efe826b3c82a1" Oct 31 20:01:30.898323 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.898296 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h"] Oct 31 20:01:30.905025 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:30.905003 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-wlc7h"] Oct 31 20:01:31.519505 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:31.519474 4910 scope.go:122] "RemoveContainer" containerID="7b4454198ba5114c1cfffcf978f97b69540e07ca2a01681c3d17ab904d9d8051" Oct 31 20:01:31.765705 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:31.765668 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89529dc6-ab2a-4b03-a4a9-56f43c12fc31" path="/var/lib/kubelet/pods/89529dc6-ab2a-4b03-a4a9-56f43c12fc31/volumes" Oct 31 20:01:35.617269 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:35.617240 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq"] Oct 31 20:01:35.627600 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:35.618129 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" podUID="67aa01d5-e453-4154-88f3-ef247ff5949e" containerName="istio-proxy" containerID="cri-o://c17d01aff8aa897d15f86c63aec420d3290295427ebfc647848f37c4333f6cfe" gracePeriod=30 Oct 31 20:01:36.286323 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:36.283851 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w"] Oct 31 20:01:36.286323 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:36.286274 4910 cpu_manager.go:414] "containerMap: removing container" podUID="89529dc6-ab2a-4b03-a4a9-56f43c12fc31" containerName="istio-proxy" Oct 31 20:01:36.286323 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:36.286304 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89529dc6-ab2a-4b03-a4a9-56f43c12fc31" containerName="istio-proxy" Oct 31 20:01:36.286323 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:36.286334 4910 cpu_manager.go:414] "containerMap: removing container" podUID="bbd3e5b5-b652-4f8f-9666-2508637b9697" containerName="revoke-keys" Oct 31 20:01:36.286723 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:36.286343 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd3e5b5-b652-4f8f-9666-2508637b9697" containerName="revoke-keys" Oct 31 20:01:36.286723 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:36.286374 4910 cpu_manager.go:414] "containerMap: removing container" podUID="627b9644-f20c-4ae1-b906-f4d8f35ee8b7" containerName="revoke-keys" Oct 31 20:01:36.286723 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:36.286383 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="627b9644-f20c-4ae1-b906-f4d8f35ee8b7" containerName="revoke-keys" Oct 31 20:01:36.286723 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:36.286584 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="89529dc6-ab2a-4b03-a4a9-56f43c12fc31" containerName="istio-proxy" Oct 31 20:01:36.286723 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:36.286598 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd3e5b5-b652-4f8f-9666-2508637b9697" containerName="revoke-keys" Oct 31 20:01:36.286723 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:36.286611 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="627b9644-f20c-4ae1-b906-f4d8f35ee8b7" containerName="revoke-keys" Oct 31 20:01:36.294873 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:36.294851 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w" Oct 31 20:01:36.296848 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:36.296819 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w"] Oct 31 20:01:36.299557 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:36.299537 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 20:01:36.300847 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:36.300818 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:01:36.416930 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:36.407524 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed3877ef-e37b-46ae-b47d-824f788857c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w\" (UID: \"ed3877ef-e37b-46ae-b47d-824f788857c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w" Oct 31 20:01:36.416930 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:36.407613 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed3877ef-e37b-46ae-b47d-824f788857c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w\" (UID: \"ed3877ef-e37b-46ae-b47d-824f788857c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w" Oct 31 20:01:36.416930 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:36.407753 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmtlc\" (UniqueName: \"kubernetes.io/projected/ed3877ef-e37b-46ae-b47d-824f788857c9-kube-api-access-tmtlc\") pod \"maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w\" (UID: \"ed3877ef-e37b-46ae-b47d-824f788857c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w" Oct 31 20:01:36.509184 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:36.509145 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmtlc\" (UniqueName: \"kubernetes.io/projected/ed3877ef-e37b-46ae-b47d-824f788857c9-kube-api-access-tmtlc\") pod \"maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w\" (UID: \"ed3877ef-e37b-46ae-b47d-824f788857c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w" Oct 31 20:01:36.509731 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:36.509710 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed3877ef-e37b-46ae-b47d-824f788857c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w\" (UID: \"ed3877ef-e37b-46ae-b47d-824f788857c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w" Oct 31 20:01:36.509897 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:36.509881 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed3877ef-e37b-46ae-b47d-824f788857c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w\" (UID: \"ed3877ef-e37b-46ae-b47d-824f788857c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w" Oct 31 20:01:36.510866 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:36.510663 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed3877ef-e37b-46ae-b47d-824f788857c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w\" (UID: \"ed3877ef-e37b-46ae-b47d-824f788857c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w" Oct 31 20:01:36.513524 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:36.513502 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed3877ef-e37b-46ae-b47d-824f788857c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w\" (UID: \"ed3877ef-e37b-46ae-b47d-824f788857c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w" Oct 31 20:01:36.525686 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:36.525658 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmtlc\" (UniqueName: \"kubernetes.io/projected/ed3877ef-e37b-46ae-b47d-824f788857c9-kube-api-access-tmtlc\") pod \"maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w\" (UID: \"ed3877ef-e37b-46ae-b47d-824f788857c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w" Oct 31 20:01:36.611588 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:36.611557 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w" Oct 31 20:01:37.003574 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:37.003542 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w"] Oct 31 20:01:37.004218 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:01:37.004194 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3877ef_e37b_46ae_b47d_824f788857c9.slice/crio-bed2252b3e8d7585916d232ae7455a3b065f7e2bda6921e55a984592dc8076c2 WatchSource:0}: Error finding container bed2252b3e8d7585916d232ae7455a3b065f7e2bda6921e55a984592dc8076c2: Status 404 returned error can't find the container with id bed2252b3e8d7585916d232ae7455a3b065f7e2bda6921e55a984592dc8076c2 Oct 31 20:01:37.368939 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:37.368305 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w" event={"ID":"ed3877ef-e37b-46ae-b47d-824f788857c9","Type":"ContainerStarted","Data":"bed2252b3e8d7585916d232ae7455a3b065f7e2bda6921e55a984592dc8076c2"} Oct 31 20:01:38.386027 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:38.385987 4910 generic.go:356] "Generic (PLEG): container finished" podID="ed3877ef-e37b-46ae-b47d-824f788857c9" containerID="120ac1858acb982dcd1e9d0eab1792f30569089cd0ce5475de6ebe1c954a6033" exitCode=0 Oct 31 20:01:38.386533 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:38.386184 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w" event={"ID":"ed3877ef-e37b-46ae-b47d-824f788857c9","Type":"ContainerDied","Data":"120ac1858acb982dcd1e9d0eab1792f30569089cd0ce5475de6ebe1c954a6033"} Oct 31 20:01:39.711219 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:39.710867 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w" Oct 31 20:01:39.800107 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:39.799745 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed3877ef-e37b-46ae-b47d-824f788857c9-kube-api-access-tmtlc\" (UniqueName: \"kubernetes.io/projected/ed3877ef-e37b-46ae-b47d-824f788857c9-kube-api-access-tmtlc\") pod \"ed3877ef-e37b-46ae-b47d-824f788857c9\" (UID: \"ed3877ef-e37b-46ae-b47d-824f788857c9\") " Oct 31 20:01:39.800107 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:39.799851 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed3877ef-e37b-46ae-b47d-824f788857c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed3877ef-e37b-46ae-b47d-824f788857c9-maas-api-xks-ca\") pod \"ed3877ef-e37b-46ae-b47d-824f788857c9\" (UID: \"ed3877ef-e37b-46ae-b47d-824f788857c9\") " Oct 31 20:01:39.800107 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:39.799943 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed3877ef-e37b-46ae-b47d-824f788857c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed3877ef-e37b-46ae-b47d-824f788857c9-maas-api-openshift-ca\") pod \"ed3877ef-e37b-46ae-b47d-824f788857c9\" (UID: \"ed3877ef-e37b-46ae-b47d-824f788857c9\") " Oct 31 20:01:39.803191 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:39.803158 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed3877ef-e37b-46ae-b47d-824f788857c9-maas-api-openshift-ca" pod "ed3877ef-e37b-46ae-b47d-824f788857c9" (UID: "ed3877ef-e37b-46ae-b47d-824f788857c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:01:39.803423 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:39.803373 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed3877ef-e37b-46ae-b47d-824f788857c9-maas-api-xks-ca" pod "ed3877ef-e37b-46ae-b47d-824f788857c9" (UID: "ed3877ef-e37b-46ae-b47d-824f788857c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:01:39.809960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:39.807282 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3877ef-e37b-46ae-b47d-824f788857c9-kube-api-access-tmtlc" pod "ed3877ef-e37b-46ae-b47d-824f788857c9" (UID: "ed3877ef-e37b-46ae-b47d-824f788857c9"). InnerVolumeSpecName "kube-api-access-tmtlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:01:39.966054 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:39.965953 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed3877ef-e37b-46ae-b47d-824f788857c9-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:39.966303 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:39.966285 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmtlc\" (UniqueName: \"kubernetes.io/projected/ed3877ef-e37b-46ae-b47d-824f788857c9-kube-api-access-tmtlc\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:39.966411 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:39.966398 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed3877ef-e37b-46ae-b47d-824f788857c9-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:40.406104 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:40.406068 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w" Oct 31 20:01:40.406272 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:40.406079 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w" event={"ID":"ed3877ef-e37b-46ae-b47d-824f788857c9","Type":"ContainerDied","Data":"bed2252b3e8d7585916d232ae7455a3b065f7e2bda6921e55a984592dc8076c2"} Oct 31 20:01:40.406272 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:40.406202 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bed2252b3e8d7585916d232ae7455a3b065f7e2bda6921e55a984592dc8076c2" Oct 31 20:01:41.043554 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.043405 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 20:01:41.198923 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.196941 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-data\") pod \"67aa01d5-e453-4154-88f3-ef247ff5949e\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " Oct 31 20:01:41.198923 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.197179 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-envoy\") pod \"67aa01d5-e453-4154-88f3-ef247ff5949e\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " Oct 31 20:01:41.198923 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.197292 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-workload-certs\") pod \"67aa01d5-e453-4154-88f3-ef247ff5949e\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " Oct 31 20:01:41.198923 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.197338 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-credential-socket\") pod \"67aa01d5-e453-4154-88f3-ef247ff5949e\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " Oct 31 20:01:41.198923 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.197516 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67aa01d5-e453-4154-88f3-ef247ff5949e-kube-api-access-pfw2d\" (UniqueName: \"kubernetes.io/projected/67aa01d5-e453-4154-88f3-ef247ff5949e-kube-api-access-pfw2d\") pod \"67aa01d5-e453-4154-88f3-ef247ff5949e\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " Oct 31 20:01:41.198923 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.197621 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-token\" (UniqueName: \"kubernetes.io/projected/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-token\") pod \"67aa01d5-e453-4154-88f3-ef247ff5949e\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " Oct 31 20:01:41.198923 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.197680 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-workload-socket\") pod \"67aa01d5-e453-4154-88f3-ef247ff5949e\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " Oct 31 20:01:41.198923 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.197768 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-podinfo\") pod \"67aa01d5-e453-4154-88f3-ef247ff5949e\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " Oct 31 20:01:41.198923 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.197829 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67aa01d5-e453-4154-88f3-ef247ff5949e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67aa01d5-e453-4154-88f3-ef247ff5949e-istiod-ca-cert\") pod \"67aa01d5-e453-4154-88f3-ef247ff5949e\" (UID: \"67aa01d5-e453-4154-88f3-ef247ff5949e\") " Oct 31 20:01:41.198923 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.198497 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-data" pod "67aa01d5-e453-4154-88f3-ef247ff5949e" (UID: "67aa01d5-e453-4154-88f3-ef247ff5949e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:01:41.198923 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.198697 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:41.199671 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.198994 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-workload-certs" pod "67aa01d5-e453-4154-88f3-ef247ff5949e" (UID: "67aa01d5-e453-4154-88f3-ef247ff5949e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:01:41.199671 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.199241 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-workload-socket" pod "67aa01d5-e453-4154-88f3-ef247ff5949e" (UID: "67aa01d5-e453-4154-88f3-ef247ff5949e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:01:41.199671 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.199560 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67aa01d5-e453-4154-88f3-ef247ff5949e-istiod-ca-cert" pod "67aa01d5-e453-4154-88f3-ef247ff5949e" (UID: "67aa01d5-e453-4154-88f3-ef247ff5949e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:01:41.199671 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.199633 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-credential-socket" pod "67aa01d5-e453-4154-88f3-ef247ff5949e" (UID: "67aa01d5-e453-4154-88f3-ef247ff5949e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:01:41.203376 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.203350 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-envoy" pod "67aa01d5-e453-4154-88f3-ef247ff5949e" (UID: "67aa01d5-e453-4154-88f3-ef247ff5949e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:01:41.203724 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.203700 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-token" pod "67aa01d5-e453-4154-88f3-ef247ff5949e" (UID: "67aa01d5-e453-4154-88f3-ef247ff5949e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:01:41.203820 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.203725 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-podinfo" pod "67aa01d5-e453-4154-88f3-ef247ff5949e" (UID: "67aa01d5-e453-4154-88f3-ef247ff5949e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:01:41.203820 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.203734 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67aa01d5-e453-4154-88f3-ef247ff5949e-kube-api-access-pfw2d" pod "67aa01d5-e453-4154-88f3-ef247ff5949e" (UID: "67aa01d5-e453-4154-88f3-ef247ff5949e"). InnerVolumeSpecName "kube-api-access-pfw2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:01:41.300804 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.300728 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:41.300804 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.300760 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:41.300804 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.300770 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:41.300804 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.300778 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67aa01d5-e453-4154-88f3-ef247ff5949e-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:41.300804 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.300787 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:41.300804 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.300795 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:41.300804 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.300807 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67aa01d5-e453-4154-88f3-ef247ff5949e-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:41.301185 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.300816 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfw2d\" (UniqueName: \"kubernetes.io/projected/67aa01d5-e453-4154-88f3-ef247ff5949e-kube-api-access-pfw2d\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:41.419774 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.419738 4910 generic.go:356] "Generic (PLEG): container finished" podID="67aa01d5-e453-4154-88f3-ef247ff5949e" containerID="c17d01aff8aa897d15f86c63aec420d3290295427ebfc647848f37c4333f6cfe" exitCode=0 Oct 31 20:01:41.420097 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.420069 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" event={"ID":"67aa01d5-e453-4154-88f3-ef247ff5949e","Type":"ContainerDied","Data":"c17d01aff8aa897d15f86c63aec420d3290295427ebfc647848f37c4333f6cfe"} Oct 31 20:01:41.420215 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.420137 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" event={"ID":"67aa01d5-e453-4154-88f3-ef247ff5949e","Type":"ContainerDied","Data":"fef68b7b4a8abc0c2acfbcd39f12c22348364c202d6ee86c4a580388713f8333"} Oct 31 20:01:41.420215 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.420082 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq" Oct 31 20:01:41.420215 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.420165 4910 scope.go:122] "RemoveContainer" containerID="c17d01aff8aa897d15f86c63aec420d3290295427ebfc647848f37c4333f6cfe" Oct 31 20:01:41.440315 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.440286 4910 scope.go:122] "RemoveContainer" containerID="c232ac7fd865f3a8f4a69550980f2ebc3d30d8d83d6476d6e434c13949825ac9" Oct 31 20:01:41.458984 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.458949 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq"] Oct 31 20:01:41.460362 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.460341 4910 scope.go:122] "RemoveContainer" containerID="c17d01aff8aa897d15f86c63aec420d3290295427ebfc647848f37c4333f6cfe" Oct 31 20:01:41.461474 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:01:41.461337 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c17d01aff8aa897d15f86c63aec420d3290295427ebfc647848f37c4333f6cfe\": container with ID starting with c17d01aff8aa897d15f86c63aec420d3290295427ebfc647848f37c4333f6cfe not found: ID does not exist" containerID="c17d01aff8aa897d15f86c63aec420d3290295427ebfc647848f37c4333f6cfe" Oct 31 20:01:41.461474 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.461388 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c17d01aff8aa897d15f86c63aec420d3290295427ebfc647848f37c4333f6cfe"} err="failed to get container status \"c17d01aff8aa897d15f86c63aec420d3290295427ebfc647848f37c4333f6cfe\": rpc error: code = NotFound desc = could not find container \"c17d01aff8aa897d15f86c63aec420d3290295427ebfc647848f37c4333f6cfe\": container with ID starting with c17d01aff8aa897d15f86c63aec420d3290295427ebfc647848f37c4333f6cfe not found: ID does not exist" Oct 31 20:01:41.461474 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.461413 4910 scope.go:122] "RemoveContainer" containerID="c232ac7fd865f3a8f4a69550980f2ebc3d30d8d83d6476d6e434c13949825ac9" Oct 31 20:01:41.462141 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:01:41.462094 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c232ac7fd865f3a8f4a69550980f2ebc3d30d8d83d6476d6e434c13949825ac9\": container with ID starting with c232ac7fd865f3a8f4a69550980f2ebc3d30d8d83d6476d6e434c13949825ac9 not found: ID does not exist" containerID="c232ac7fd865f3a8f4a69550980f2ebc3d30d8d83d6476d6e434c13949825ac9" Oct 31 20:01:41.462246 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.462148 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c232ac7fd865f3a8f4a69550980f2ebc3d30d8d83d6476d6e434c13949825ac9"} err="failed to get container status \"c232ac7fd865f3a8f4a69550980f2ebc3d30d8d83d6476d6e434c13949825ac9\": rpc error: code = NotFound desc = could not find container \"c232ac7fd865f3a8f4a69550980f2ebc3d30d8d83d6476d6e434c13949825ac9\": container with ID starting with c232ac7fd865f3a8f4a69550980f2ebc3d30d8d83d6476d6e434c13949825ac9 not found: ID does not exist" Oct 31 20:01:41.470486 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.470451 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-26cd59-openshift-default-74bbc85d85-kk2sq"] Oct 31 20:01:41.763357 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:41.763316 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67aa01d5-e453-4154-88f3-ef247ff5949e" path="/var/lib/kubelet/pods/67aa01d5-e453-4154-88f3-ef247ff5949e/volumes" Oct 31 20:01:42.717145 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:01:42.717101 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:01:44.448427 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:44.448342 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q"] Oct 31 20:01:44.516959 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:44.498784 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bbf75-kcc4q"] Oct 31 20:01:45.068755 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.068713 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr"] Oct 31 20:01:45.069771 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.069745 4910 cpu_manager.go:414] "containerMap: removing container" podUID="67aa01d5-e453-4154-88f3-ef247ff5949e" containerName="istio-proxy" Oct 31 20:01:45.069883 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.069774 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67aa01d5-e453-4154-88f3-ef247ff5949e" containerName="istio-proxy" Oct 31 20:01:45.069883 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.069852 4910 cpu_manager.go:414] "containerMap: removing container" podUID="ed3877ef-e37b-46ae-b47d-824f788857c9" containerName="revoke-keys" Oct 31 20:01:45.069883 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.069860 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed3877ef-e37b-46ae-b47d-824f788857c9" containerName="revoke-keys" Oct 31 20:01:45.070087 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.069885 4910 cpu_manager.go:414] "containerMap: removing container" podUID="67aa01d5-e453-4154-88f3-ef247ff5949e" containerName="istio-proxy" Oct 31 20:01:45.070087 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.069894 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67aa01d5-e453-4154-88f3-ef247ff5949e" containerName="istio-proxy" Oct 31 20:01:45.070188 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.070091 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="67aa01d5-e453-4154-88f3-ef247ff5949e" containerName="istio-proxy" Oct 31 20:01:45.070188 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.070122 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed3877ef-e37b-46ae-b47d-824f788857c9" containerName="revoke-keys" Oct 31 20:01:45.076738 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.076713 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.130241 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.119113 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr"] Oct 31 20:01:45.163173 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.163132 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687609ff-2575-418e-80e9-c02f7550a362-istiod-ca-cert\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.163366 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.163200 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-workload-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.163366 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.163304 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-credential-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.163478 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.163424 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-workload-certs\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.163521 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.163485 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5l7w\" (UniqueName: \"kubernetes.io/projected/687609ff-2575-418e-80e9-c02f7550a362-kube-api-access-t5l7w\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.163606 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.163589 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-istio-envoy\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.163678 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.163663 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-istio-data\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.163741 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.163711 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687609ff-2575-418e-80e9-c02f7550a362-istio-podinfo\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.163976 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.163959 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687609ff-2575-418e-80e9-c02f7550a362-istio-token\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.265558 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.265531 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687609ff-2575-418e-80e9-c02f7550a362-istio-podinfo\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.265702 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.265690 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687609ff-2575-418e-80e9-c02f7550a362-istio-token\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.265766 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.265754 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687609ff-2575-418e-80e9-c02f7550a362-istiod-ca-cert\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.265811 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.265788 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-workload-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.265861 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.265820 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-credential-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.265939 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.265882 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-workload-certs\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.266007 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.265940 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5l7w\" (UniqueName: \"kubernetes.io/projected/687609ff-2575-418e-80e9-c02f7550a362-kube-api-access-t5l7w\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.266070 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.266008 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-istio-envoy\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.266070 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.266050 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-istio-data\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.266309 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.266284 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-workload-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.266393 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.266327 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-workload-certs\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.266504 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.266481 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-istio-data\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.266661 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.266581 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-credential-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.266759 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.266605 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687609ff-2575-418e-80e9-c02f7550a362-istiod-ca-cert\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.268562 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.268540 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687609ff-2575-418e-80e9-c02f7550a362-istio-podinfo\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.269114 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.269087 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-istio-envoy\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.291327 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.291298 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687609ff-2575-418e-80e9-c02f7550a362-istio-token\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.292766 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.292737 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5l7w\" (UniqueName: \"kubernetes.io/projected/687609ff-2575-418e-80e9-c02f7550a362-kube-api-access-t5l7w\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.395877 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.395534 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:45.544833 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.544492 4910 generic.go:356] "Generic (PLEG): container finished" podID="d1110e52-d926-449c-809b-be04b7d1db94" containerID="e64cde6988e8d049373a6e609b79a0f2e62a4d924b627bb3e04fd1f1f97ee053" exitCode=0 Oct 31 20:01:45.544833 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.544687 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" event={"ID":"d1110e52-d926-449c-809b-be04b7d1db94","Type":"ContainerDied","Data":"e64cde6988e8d049373a6e609b79a0f2e62a4d924b627bb3e04fd1f1f97ee053"} Oct 31 20:01:45.545424 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.545276 4910 scope.go:122] "RemoveContainer" containerID="e64cde6988e8d049373a6e609b79a0f2e62a4d924b627bb3e04fd1f1f97ee053" Oct 31 20:01:45.548049 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.547772 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:01:45.548049 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.547854 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:01:45.548049 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.547887 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:01:45.787951 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.784900 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:01:45.787951 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.785092 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:01:45.787951 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.785178 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:01:45.809172 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.809108 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fc2febc-bb66-461f-86d1-9144350e0f5d" path="/var/lib/kubelet/pods/6fc2febc-bb66-461f-86d1-9144350e0f5d/volumes" Oct 31 20:01:45.810117 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:45.810092 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr"] Oct 31 20:01:46.603774 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:46.603733 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" event={"ID":"d1110e52-d926-449c-809b-be04b7d1db94","Type":"ContainerStarted","Data":"960412caed823b7ab7ba1f5b80dfa9b6a2af2343fa882b8b853457ce1d24b363"} Oct 31 20:01:46.614041 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:46.613995 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" event={"ID":"687609ff-2575-418e-80e9-c02f7550a362","Type":"ContainerStarted","Data":"1ede0bf5303e5c793ec3efe82ec21493974a183a3e95567207672227b52116e7"} Oct 31 20:01:46.614307 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:46.614286 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" event={"ID":"687609ff-2575-418e-80e9-c02f7550a362","Type":"ContainerStarted","Data":"b7540d0ea54ef9d441c1292301fdb2a0d6c96530d0335b32ca4ce5f6144fb54b"} Oct 31 20:01:46.700541 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:46.700363 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" podStartSLOduration=1.700348448 podStartE2EDuration="1.700348448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:01:46.683356081 +0000 UTC m=+2541.046176453" watchObservedRunningTime="2026-10-31 20:01:46.700348448 +0000 UTC m=+2541.063168819" Oct 31 20:01:47.184743 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:47.184708 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:01:47.186353 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:47.186326 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 31 20:01:47.186535 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:47.186510 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" podUID="d1110e52-d926-449c-809b-be04b7d1db94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 31 20:01:47.398932 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:47.397716 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:47.409944 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:47.399435 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Oct 31 20:01:47.409944 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:47.399495 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" podUID="687609ff-2575-418e-80e9-c02f7550a362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Oct 31 20:01:48.038938 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:48.038023 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:01:48.186722 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:48.186681 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 31 20:01:48.186900 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:48.186768 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" podUID="d1110e52-d926-449c-809b-be04b7d1db94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 31 20:01:48.401201 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:48.401169 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Oct 31 20:01:48.401463 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:48.401436 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" podUID="687609ff-2575-418e-80e9-c02f7550a362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Oct 31 20:01:49.185432 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:49.185386 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 31 20:01:49.186017 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:49.185470 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" podUID="d1110e52-d926-449c-809b-be04b7d1db94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 31 20:01:49.397670 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:49.396500 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Oct 31 20:01:49.397670 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:49.396547 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" podUID="687609ff-2575-418e-80e9-c02f7550a362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Oct 31 20:01:50.200935 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:50.194021 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 31 20:01:50.200935 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:50.194111 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" podUID="d1110e52-d926-449c-809b-be04b7d1db94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 31 20:01:50.402962 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:50.399110 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Oct 31 20:01:50.402962 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:50.399187 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" podUID="687609ff-2575-418e-80e9-c02f7550a362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Oct 31 20:01:51.185935 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:51.185387 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 31 20:01:51.185935 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:51.185437 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" podUID="d1110e52-d926-449c-809b-be04b7d1db94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 31 20:01:51.398222 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:51.398175 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Oct 31 20:01:51.398693 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:51.398249 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" podUID="687609ff-2575-418e-80e9-c02f7550a362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Oct 31 20:01:52.270318 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:52.270280 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 31 20:01:52.270589 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:52.270561 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" podUID="d1110e52-d926-449c-809b-be04b7d1db94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 31 20:01:52.325879 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:52.322458 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j"] Oct 31 20:01:52.325879 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:52.323703 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="67aa01d5-e453-4154-88f3-ef247ff5949e" containerName="istio-proxy" Oct 31 20:01:52.410440 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:52.410403 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j"] Oct 31 20:01:52.411113 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:52.411096 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j" Oct 31 20:01:52.431934 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:52.415898 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zstlb\" (UniqueName: \"kubernetes.io/projected/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-kube-api-access-zstlb\") pod \"maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j\" (UID: \"2358e6a2-66b8-44b0-99b6-52b9c75e2cfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j" Oct 31 20:01:52.431934 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:52.416106 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j\" (UID: \"2358e6a2-66b8-44b0-99b6-52b9c75e2cfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j" Oct 31 20:01:52.431934 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:52.416270 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-metrics-certs\") pod \"maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j\" (UID: \"2358e6a2-66b8-44b0-99b6-52b9c75e2cfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j" Oct 31 20:01:52.431934 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:52.416594 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j\" (UID: \"2358e6a2-66b8-44b0-99b6-52b9c75e2cfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j" Oct 31 20:01:52.469581 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:52.469546 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Oct 31 20:01:52.469882 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:52.469848 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" podUID="687609ff-2575-418e-80e9-c02f7550a362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Oct 31 20:01:52.472475 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:52.472450 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c9b9a1-serving-cert\"" Oct 31 20:01:52.491546 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:52.491507 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs"] Oct 31 20:01:52.491981 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:52.491943 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" podUID="b12cfe0e-fab7-4a92-b62c-a18a3771bb1a" containerName="istio-proxy" containerID="cri-o://bf5ad597b7cd5f39d29c2bccd0ea5cbf4971142f6438f85d22b568c54437cd66" gracePeriod=30 Oct 31 20:01:52.518141 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:52.518107 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j\" (UID: \"2358e6a2-66b8-44b0-99b6-52b9c75e2cfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j" Oct 31 20:01:52.518403 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:52.518378 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zstlb\" (UniqueName: \"kubernetes.io/projected/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-kube-api-access-zstlb\") pod \"maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j\" (UID: \"2358e6a2-66b8-44b0-99b6-52b9c75e2cfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j" Oct 31 20:01:52.518641 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:52.518618 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j\" (UID: \"2358e6a2-66b8-44b0-99b6-52b9c75e2cfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j" Oct 31 20:01:52.518847 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:52.518829 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-metrics-certs\") pod \"maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j\" (UID: \"2358e6a2-66b8-44b0-99b6-52b9c75e2cfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j" Oct 31 20:01:52.524031 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:52.523974 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j\" (UID: \"2358e6a2-66b8-44b0-99b6-52b9c75e2cfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j" Oct 31 20:01:52.534933 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:52.533677 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j\" (UID: \"2358e6a2-66b8-44b0-99b6-52b9c75e2cfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j" Oct 31 20:01:52.575538 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:52.575504 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-metrics-certs\") pod \"maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j\" (UID: \"2358e6a2-66b8-44b0-99b6-52b9c75e2cfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j" Oct 31 20:01:52.579665 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:52.579594 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zstlb\" (UniqueName: \"kubernetes.io/projected/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-kube-api-access-zstlb\") pod \"maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j\" (UID: \"2358e6a2-66b8-44b0-99b6-52b9c75e2cfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j" Oct 31 20:01:52.586056 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:52.586019 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j" Oct 31 20:01:53.723018 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.689899 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt"] Oct 31 20:01:53.790541 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.772528 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:53.790541 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.778242 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d7647c-openshift-default-dockercfg-jm4k9\"" Oct 31 20:01:53.826933 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.826142 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt"] Oct 31 20:01:53.826933 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.826198 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j"] Oct 31 20:01:53.909480 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.895281 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-envoy\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:53.909480 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.895330 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-workload-socket\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:53.909480 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.895457 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istiod-ca-cert\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:53.909480 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.895521 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-workload-certs\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:53.909480 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.895604 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-credential-socket\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:53.909480 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.895646 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfhzl\" (UniqueName: \"kubernetes.io/projected/33ed4412-c59e-4ea3-87cd-5d5b485ade85-kube-api-access-tfhzl\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:53.909480 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.895795 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-podinfo\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:53.909480 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.895827 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-data\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:53.909480 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.895903 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-token\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:53.968313 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.968279 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:53.972856 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.972372 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:53.978159 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.978092 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:01:53.992808 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.992422 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:01:53.995787 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.995764 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:01:53.997374 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.997337 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-envoy\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:53.997554 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.997536 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-workload-socket\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:53.997714 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.997700 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istiod-ca-cert\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:53.997799 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.997790 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-workload-certs\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:53.998659 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.998638 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-credential-socket\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:53.998845 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.998827 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfhzl\" (UniqueName: \"kubernetes.io/projected/33ed4412-c59e-4ea3-87cd-5d5b485ade85-kube-api-access-tfhzl\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:53.999076 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.999057 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-podinfo\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:53.999528 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.999509 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-data\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:54.000032 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:54.000012 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-token\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:54.000453 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.998317 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-workload-certs\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:54.000579 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.999900 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-data\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:54.000660 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.999444 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-credential-socket\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:54.000748 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:53.998437 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-workload-socket\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:54.000870 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:54.000624 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istiod-ca-cert\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:54.004763 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:54.004733 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-envoy\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:54.028984 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:54.027642 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-podinfo\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:54.054645 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:54.043978 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-token\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:54.054645 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:54.045644 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfhzl\" (UniqueName: \"kubernetes.io/projected/33ed4412-c59e-4ea3-87cd-5d5b485ade85-kube-api-access-tfhzl\") pod \"e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:54.206113 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:54.206072 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:54.612471 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:01:54.612077 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ed4412_c59e_4ea3_87cd_5d5b485ade85.slice/crio-d47686d5f902e9a136a3b6e5d3493fd9dd480765eabc49b0d7e917bf963788d0 WatchSource:0}: Error finding container d47686d5f902e9a136a3b6e5d3493fd9dd480765eabc49b0d7e917bf963788d0: Status 404 returned error can't find the container with id d47686d5f902e9a136a3b6e5d3493fd9dd480765eabc49b0d7e917bf963788d0 Oct 31 20:01:54.614846 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:54.614819 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt"] Oct 31 20:01:54.615824 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:54.615782 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:01:54.615975 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:54.615878 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:01:54.615975 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:54.615951 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:01:54.765571 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:54.765392 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j" event={"ID":"2358e6a2-66b8-44b0-99b6-52b9c75e2cfe","Type":"ContainerStarted","Data":"dadaf4e0f0739e690f91dbc604e2fbda5d47a968d5f4b1fba29967a63cf6e8aa"} Oct 31 20:01:54.765571 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:54.765456 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j" event={"ID":"2358e6a2-66b8-44b0-99b6-52b9c75e2cfe","Type":"ContainerStarted","Data":"f205c15c0265abeb91a5f0d98125c391bda43c21f02258bca9ed75e0b496e250"} Oct 31 20:01:54.765571 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:54.765534 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j" Oct 31 20:01:54.768241 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:54.768209 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" event={"ID":"33ed4412-c59e-4ea3-87cd-5d5b485ade85","Type":"ContainerStarted","Data":"2028ce63f101002960ddc8aa55cb6add8487cf67ee7e06a39880abc0cceba943"} Oct 31 20:01:54.768449 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:54.768432 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" event={"ID":"33ed4412-c59e-4ea3-87cd-5d5b485ade85","Type":"ContainerStarted","Data":"d47686d5f902e9a136a3b6e5d3493fd9dd480765eabc49b0d7e917bf963788d0"} Oct 31 20:01:54.796773 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:54.796721 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j" podStartSLOduration=2.261380865 podStartE2EDuration="2.796705428s" totalImagesPullingTime="535.324563ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:01:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:01:54.791124897 +0000 UTC m=+2549.153945269" watchObservedRunningTime="2026-10-31 20:01:54.796705428 +0000 UTC m=+2549.159525803" Oct 31 20:01:54.823189 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:54.823127 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" podStartSLOduration=1.823106253 podStartE2EDuration="1.823106253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:01:54.821030955 +0000 UTC m=+2549.183851327" watchObservedRunningTime="2026-10-31 20:01:54.823106253 +0000 UTC m=+2549.185926627" Oct 31 20:01:55.206659 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:55.206613 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:55.208235 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:55.208196 4910 patch_prober.go:29] interesting pod/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 31 20:01:55.208382 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:55.208263 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" podUID="33ed4412-c59e-4ea3-87cd-5d5b485ade85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 31 20:01:56.212460 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:56.212425 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:56.794454 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:56.794419 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:56.795570 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:56.795546 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:01:57.806650 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.806621 4910 generic.go:356] "Generic (PLEG): container finished" podID="b12cfe0e-fab7-4a92-b62c-a18a3771bb1a" containerID="bf5ad597b7cd5f39d29c2bccd0ea5cbf4971142f6438f85d22b568c54437cd66" exitCode=0 Oct 31 20:01:57.807068 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.806695 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" event={"ID":"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a","Type":"ContainerDied","Data":"bf5ad597b7cd5f39d29c2bccd0ea5cbf4971142f6438f85d22b568c54437cd66"} Oct 31 20:01:57.807068 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.806761 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" event={"ID":"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a","Type":"ContainerDied","Data":"b78655a261267444c050882b766b4b20a7c8f14dc2f313f6aee83f9bdca0b69c"} Oct 31 20:01:57.807068 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.806793 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b78655a261267444c050882b766b4b20a7c8f14dc2f313f6aee83f9bdca0b69c" Oct 31 20:01:57.825454 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.825387 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 20:01:57.954923 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.954879 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-credential-socket\") pod \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " Oct 31 20:01:57.955122 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.954935 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-token\" (UniqueName: \"kubernetes.io/projected/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-token\") pod \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " Oct 31 20:01:57.955122 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.954993 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-podinfo\") pod \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " Oct 31 20:01:57.955122 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.955014 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-data\") pod \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " Oct 31 20:01:57.955319 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.955149 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istiod-ca-cert\") pod \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " Oct 31 20:01:57.955319 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.955208 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-workload-socket\") pod \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " Oct 31 20:01:57.955319 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.955220 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-credential-socket" pod "b12cfe0e-fab7-4a92-b62c-a18a3771bb1a" (UID: "b12cfe0e-fab7-4a92-b62c-a18a3771bb1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:01:57.955319 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.955280 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-workload-certs\") pod \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " Oct 31 20:01:57.955603 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.955342 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-data" pod "b12cfe0e-fab7-4a92-b62c-a18a3771bb1a" (UID: "b12cfe0e-fab7-4a92-b62c-a18a3771bb1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:01:57.955603 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.955377 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-kube-api-access-d9dlr\" (UniqueName: \"kubernetes.io/projected/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-kube-api-access-d9dlr\") pod \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " Oct 31 20:01:57.955603 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.955454 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-envoy\") pod \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\" (UID: \"b12cfe0e-fab7-4a92-b62c-a18a3771bb1a\") " Oct 31 20:01:57.955603 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.955556 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-workload-socket" pod "b12cfe0e-fab7-4a92-b62c-a18a3771bb1a" (UID: "b12cfe0e-fab7-4a92-b62c-a18a3771bb1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:01:57.955603 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.955570 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istiod-ca-cert" pod "b12cfe0e-fab7-4a92-b62c-a18a3771bb1a" (UID: "b12cfe0e-fab7-4a92-b62c-a18a3771bb1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:01:57.955892 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.955757 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-workload-certs" pod "b12cfe0e-fab7-4a92-b62c-a18a3771bb1a" (UID: "b12cfe0e-fab7-4a92-b62c-a18a3771bb1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:01:57.956773 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.956744 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:57.956773 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.956774 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:57.957036 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.956790 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:57.957036 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.956804 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:57.957036 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.956817 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:57.957984 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.957955 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-podinfo" pod "b12cfe0e-fab7-4a92-b62c-a18a3771bb1a" (UID: "b12cfe0e-fab7-4a92-b62c-a18a3771bb1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:01:57.958103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.958000 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-kube-api-access-d9dlr" pod "b12cfe0e-fab7-4a92-b62c-a18a3771bb1a" (UID: "b12cfe0e-fab7-4a92-b62c-a18a3771bb1a"). InnerVolumeSpecName "kube-api-access-d9dlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:01:57.958103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.958032 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-token" pod "b12cfe0e-fab7-4a92-b62c-a18a3771bb1a" (UID: "b12cfe0e-fab7-4a92-b62c-a18a3771bb1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:01:57.958245 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:57.958228 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-envoy" pod "b12cfe0e-fab7-4a92-b62c-a18a3771bb1a" (UID: "b12cfe0e-fab7-4a92-b62c-a18a3771bb1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:01:58.058453 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:58.058417 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:58.058453 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:58.058453 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:58.058663 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:58.058468 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9dlr\" (UniqueName: \"kubernetes.io/projected/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-kube-api-access-d9dlr\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:58.058663 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:58.058482 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:01:58.867932 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:58.867876 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs" Oct 31 20:01:58.984954 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:58.984894 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs"] Oct 31 20:01:58.998368 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:58.998337 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-h89zs"] Oct 31 20:01:59.758268 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:01:59.758224 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b12cfe0e-fab7-4a92-b62c-a18a3771bb1a" path="/var/lib/kubelet/pods/b12cfe0e-fab7-4a92-b62c-a18a3771bb1a/volumes" Oct 31 20:02:00.793133 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:00.793100 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j" Oct 31 20:02:07.922103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:07.917980 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w"] Oct 31 20:02:07.929780 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:07.923324 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5f3eff-lm92w"] Oct 31 20:02:09.756631 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:09.756592 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed3877ef-e37b-46ae-b47d-824f788857c9" path="/var/lib/kubelet/pods/ed3877ef-e37b-46ae-b47d-824f788857c9/volumes" Oct 31 20:02:12.729146 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:02:12.729101 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:02:15.163022 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.162989 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc"] Oct 31 20:02:15.164639 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.164315 4910 cpu_manager.go:414] "containerMap: removing container" podUID="b12cfe0e-fab7-4a92-b62c-a18a3771bb1a" containerName="istio-proxy" Oct 31 20:02:15.165187 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.165162 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b12cfe0e-fab7-4a92-b62c-a18a3771bb1a" containerName="istio-proxy" Oct 31 20:02:15.165751 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.165611 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="b12cfe0e-fab7-4a92-b62c-a18a3771bb1a" containerName="istio-proxy" Oct 31 20:02:15.177782 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.177752 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.244159 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.220967 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc"] Oct 31 20:02:15.247103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.245923 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrsmv\" (UniqueName: \"kubernetes.io/projected/94f330fb-bcfb-4a9d-9331-48a619009f36-kube-api-access-zrsmv\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.247103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.246075 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-workload-certs\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.247103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.246108 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-data\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.247103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.246157 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-credential-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.247103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.246186 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-podinfo\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.247103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.246234 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-envoy\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.247103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.246378 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f330fb-bcfb-4a9d-9331-48a619009f36-istiod-ca-cert\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.247103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.246412 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-workload-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.247103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.246496 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-token\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.347641 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.347604 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f330fb-bcfb-4a9d-9331-48a619009f36-istiod-ca-cert\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.347641 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.347654 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-workload-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.347930 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.347707 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-token\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.347930 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.347823 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrsmv\" (UniqueName: \"kubernetes.io/projected/94f330fb-bcfb-4a9d-9331-48a619009f36-kube-api-access-zrsmv\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.347930 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.347927 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-workload-certs\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.348115 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.347952 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-data\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.348115 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.347999 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-credential-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.348115 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.348029 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-podinfo\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.348115 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.348076 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-envoy\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.348389 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.348134 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-workload-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.348389 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.348297 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-workload-certs\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.348556 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.348533 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f330fb-bcfb-4a9d-9331-48a619009f36-istiod-ca-cert\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.348701 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.348546 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-data\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.348764 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.348724 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-credential-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.350573 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.350549 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-envoy\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.350714 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.350697 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-podinfo\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.360819 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.360783 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-token\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.361070 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.361054 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrsmv\" (UniqueName: \"kubernetes.io/projected/94f330fb-bcfb-4a9d-9331-48a619009f36-kube-api-access-zrsmv\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.521360 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.521230 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:15.868431 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:02:15.868369 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f330fb_bcfb_4a9d_9331_48a619009f36.slice/crio-6df69d801445fd98cde9f358bfb2d39a847b72671cc76c8d104d0789504cdf8b WatchSource:0}: Error finding container 6df69d801445fd98cde9f358bfb2d39a847b72671cc76c8d104d0789504cdf8b: Status 404 returned error can't find the container with id 6df69d801445fd98cde9f358bfb2d39a847b72671cc76c8d104d0789504cdf8b Oct 31 20:02:15.871116 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.871035 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc"] Oct 31 20:02:15.872134 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.871982 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:02:15.872134 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.872085 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:02:15.872386 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:15.872157 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:02:16.135819 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:16.135732 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" event={"ID":"94f330fb-bcfb-4a9d-9331-48a619009f36","Type":"ContainerStarted","Data":"30a881d040a7d7703a5253455c8ce0c06a357f9b6f2ff7f2a026e79515a11069"} Oct 31 20:02:16.137058 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:16.137020 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" event={"ID":"94f330fb-bcfb-4a9d-9331-48a619009f36","Type":"ContainerStarted","Data":"6df69d801445fd98cde9f358bfb2d39a847b72671cc76c8d104d0789504cdf8b"} Oct 31 20:02:16.163310 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:16.163224 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" podStartSLOduration=1.1632056259999999 podStartE2EDuration="1.163205626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:02:16.16057813 +0000 UTC m=+2570.523398514" watchObservedRunningTime="2026-10-31 20:02:16.163205626 +0000 UTC m=+2570.526025998" Oct 31 20:02:16.525999 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:16.523674 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:16.525999 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:16.525505 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 31 20:02:16.525999 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:16.525574 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" podUID="94f330fb-bcfb-4a9d-9331-48a619009f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 31 20:02:17.531939 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:17.531831 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 31 20:02:17.534267 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:17.531893 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" podUID="94f330fb-bcfb-4a9d-9331-48a619009f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 31 20:02:18.529975 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:18.524369 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 31 20:02:18.529975 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:18.524477 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" podUID="94f330fb-bcfb-4a9d-9331-48a619009f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 31 20:02:19.542936 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:19.530048 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 31 20:02:19.542936 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:19.530116 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" podUID="94f330fb-bcfb-4a9d-9331-48a619009f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 31 20:02:19.996464 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:19.996423 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4"] Oct 31 20:02:20.002679 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:20.002650 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" Oct 31 20:02:20.041939 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:20.039007 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d7647c-serving-cert\"" Oct 31 20:02:20.065018 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:20.063967 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22f35b72-e954-4d06-ab62-e7a5502a9929-metrics-certs\") pod \"maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4\" (UID: \"22f35b72-e954-4d06-ab62-e7a5502a9929\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" Oct 31 20:02:20.065018 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:20.064019 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22f35b72-e954-4d06-ab62-e7a5502a9929-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4\" (UID: \"22f35b72-e954-4d06-ab62-e7a5502a9929\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" Oct 31 20:02:20.065018 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:20.064103 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgsgn\" (UniqueName: \"kubernetes.io/projected/22f35b72-e954-4d06-ab62-e7a5502a9929-kube-api-access-tgsgn\") pod \"maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4\" (UID: \"22f35b72-e954-4d06-ab62-e7a5502a9929\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" Oct 31 20:02:20.065018 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:20.064193 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22f35b72-e954-4d06-ab62-e7a5502a9929-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4\" (UID: \"22f35b72-e954-4d06-ab62-e7a5502a9929\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" Oct 31 20:02:20.066453 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:20.066426 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4"] Oct 31 20:02:20.190093 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:20.165772 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgsgn\" (UniqueName: \"kubernetes.io/projected/22f35b72-e954-4d06-ab62-e7a5502a9929-kube-api-access-tgsgn\") pod \"maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4\" (UID: \"22f35b72-e954-4d06-ab62-e7a5502a9929\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" Oct 31 20:02:20.190543 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:20.190513 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22f35b72-e954-4d06-ab62-e7a5502a9929-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4\" (UID: \"22f35b72-e954-4d06-ab62-e7a5502a9929\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" Oct 31 20:02:20.191122 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:20.191091 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22f35b72-e954-4d06-ab62-e7a5502a9929-metrics-certs\") pod \"maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4\" (UID: \"22f35b72-e954-4d06-ab62-e7a5502a9929\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" Oct 31 20:02:20.198936 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:20.191369 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22f35b72-e954-4d06-ab62-e7a5502a9929-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4\" (UID: \"22f35b72-e954-4d06-ab62-e7a5502a9929\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" Oct 31 20:02:20.199401 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:02:20.192084 4910 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-serving-cert: secret "maas-api-e2e-ipp-b-d7647c-serving-cert" not found Oct 31 20:02:20.202312 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:02:20.202286 4910 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22f35b72-e954-4d06-ab62-e7a5502a9929-maas-api-tls podName:22f35b72-e954-4d06-ab62-e7a5502a9929 nodeName:}" failed. No retries permitted until 2026-10-31 20:02:20.70225509 +0000 UTC m=+2575.065075452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/22f35b72-e954-4d06-ab62-e7a5502a9929-maas-api-tls") pod "maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" (UID: "22f35b72-e954-4d06-ab62-e7a5502a9929") : secret "maas-api-e2e-ipp-b-d7647c-serving-cert" not found Oct 31 20:02:20.207011 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:20.193717 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22f35b72-e954-4d06-ab62-e7a5502a9929-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4\" (UID: \"22f35b72-e954-4d06-ab62-e7a5502a9929\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" Oct 31 20:02:20.222999 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:20.216334 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgsgn\" (UniqueName: \"kubernetes.io/projected/22f35b72-e954-4d06-ab62-e7a5502a9929-kube-api-access-tgsgn\") pod \"maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4\" (UID: \"22f35b72-e954-4d06-ab62-e7a5502a9929\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" Oct 31 20:02:20.226936 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:20.223665 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22f35b72-e954-4d06-ab62-e7a5502a9929-metrics-certs\") pod \"maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4\" (UID: \"22f35b72-e954-4d06-ab62-e7a5502a9929\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" Oct 31 20:02:20.558936 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:20.555143 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 31 20:02:20.558936 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:20.555430 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" podUID="94f330fb-bcfb-4a9d-9331-48a619009f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 31 20:02:20.728940 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:20.720018 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22f35b72-e954-4d06-ab62-e7a5502a9929-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4\" (UID: \"22f35b72-e954-4d06-ab62-e7a5502a9929\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" Oct 31 20:02:20.847997 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:20.845724 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22f35b72-e954-4d06-ab62-e7a5502a9929-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4\" (UID: \"22f35b72-e954-4d06-ab62-e7a5502a9929\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" Oct 31 20:02:20.976945 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:20.975351 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" Oct 31 20:02:21.392596 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:02:21.392563 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f35b72_e954_4d06_ab62_e7a5502a9929.slice/crio-42c8bcd29055c2d0d0a8f0b6679140faf78aa26d494c0708010af25403350548 WatchSource:0}: Error finding container 42c8bcd29055c2d0d0a8f0b6679140faf78aa26d494c0708010af25403350548: Status 404 returned error can't find the container with id 42c8bcd29055c2d0d0a8f0b6679140faf78aa26d494c0708010af25403350548 Oct 31 20:02:21.396423 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:21.396394 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4"] Oct 31 20:02:21.545427 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:21.543141 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 31 20:02:21.545427 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:21.543224 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" podUID="94f330fb-bcfb-4a9d-9331-48a619009f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 31 20:02:22.314495 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:22.314405 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" event={"ID":"22f35b72-e954-4d06-ab62-e7a5502a9929","Type":"ContainerStarted","Data":"b552b386a0267a1c038217ee045fde6b9e1d1657b712535247f30c40f7604e77"} Oct 31 20:02:22.314495 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:22.314468 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" event={"ID":"22f35b72-e954-4d06-ab62-e7a5502a9929","Type":"ContainerStarted","Data":"42c8bcd29055c2d0d0a8f0b6679140faf78aa26d494c0708010af25403350548"} Oct 31 20:02:22.314495 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:22.314498 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" Oct 31 20:02:22.418936 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:22.406425 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" podStartSLOduration=2.801248464 podStartE2EDuration="3.406401867s" totalImagesPullingTime="605.153403ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:02:22.351455821 +0000 UTC m=+2576.714276192" watchObservedRunningTime="2026-10-31 20:02:22.406401867 +0000 UTC m=+2576.769222221" Oct 31 20:02:22.529990 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:22.525302 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body= Oct 31 20:02:22.529990 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:22.525363 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" podUID="94f330fb-bcfb-4a9d-9331-48a619009f36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" Oct 31 20:02:23.531108 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:23.531040 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:23.533101 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:23.531879 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:23.533101 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:23.532615 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:02:28.333558 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:28.333525 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" Oct 31 20:02:31.576425 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:31.576387 4910 scope.go:122] "RemoveContainer" containerID="bf5ad597b7cd5f39d29c2bccd0ea5cbf4971142f6438f85d22b568c54437cd66" Oct 31 20:02:31.588137 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:31.588116 4910 scope.go:122] "RemoveContainer" containerID="3ebe502147ee80ed5b815939868fa8a7c0e358de4d30da7be1ad2806eeba84b2" Oct 31 20:02:31.597967 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:31.597946 4910 scope.go:122] "RemoveContainer" containerID="802f0797e77955c3ecab5b994f28424afa35f2dffc5025b674bb77a1bc733a31" Oct 31 20:02:31.611177 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:31.610844 4910 scope.go:122] "RemoveContainer" containerID="44d2f91e56c72b22210a545fa6abe640c7440d95c70b521b23b6a00ea845f6d9" Oct 31 20:02:36.451767 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:36.450479 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv"] Oct 31 20:02:36.451767 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:36.450750 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" podUID="d2cd527e-0cf0-4513-97b6-7f1f9d8d1584" containerName="istio-proxy" containerID="cri-o://b390e60d26d488396a2d95fc5d1f493fd90143d1a0fe8aa9dc7680bb0d189b5f" gracePeriod=30 Oct 31 20:02:38.420249 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:38.420203 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:02:38.420630 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:38.420276 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" podUID="d2cd527e-0cf0-4513-97b6-7f1f9d8d1584" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:02:41.661422 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.661370 4910 generic.go:356] "Generic (PLEG): container finished" podID="d2cd527e-0cf0-4513-97b6-7f1f9d8d1584" containerID="b390e60d26d488396a2d95fc5d1f493fd90143d1a0fe8aa9dc7680bb0d189b5f" exitCode=0 Oct 31 20:02:41.664713 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.663807 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" event={"ID":"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584","Type":"ContainerDied","Data":"b390e60d26d488396a2d95fc5d1f493fd90143d1a0fe8aa9dc7680bb0d189b5f"} Oct 31 20:02:41.764719 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.764688 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 20:02:41.866524 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.866489 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-data\") pod \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " Oct 31 20:02:41.866732 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.866580 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-credential-socket\") pod \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " Oct 31 20:02:41.866732 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.866645 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-workload-socket\") pod \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " Oct 31 20:02:41.866732 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.866714 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-podinfo\") pod \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " Oct 31 20:02:41.866902 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.866767 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istiod-ca-cert\") pod \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " Oct 31 20:02:41.866991 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.866816 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-data" pod "d2cd527e-0cf0-4513-97b6-7f1f9d8d1584" (UID: "d2cd527e-0cf0-4513-97b6-7f1f9d8d1584"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:41.866991 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.866927 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-workload-certs\") pod \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " Oct 31 20:02:41.867114 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.866991 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-credential-socket" pod "d2cd527e-0cf0-4513-97b6-7f1f9d8d1584" (UID: "d2cd527e-0cf0-4513-97b6-7f1f9d8d1584"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:41.867114 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.867091 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-kube-api-access-mbjbx\" (UniqueName: \"kubernetes.io/projected/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-kube-api-access-mbjbx\") pod \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " Oct 31 20:02:41.867231 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.867204 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-workload-socket" pod "d2cd527e-0cf0-4513-97b6-7f1f9d8d1584" (UID: "d2cd527e-0cf0-4513-97b6-7f1f9d8d1584"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:41.867291 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.867240 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-token\" (UniqueName: \"kubernetes.io/projected/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-token\") pod \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " Oct 31 20:02:41.867344 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.867285 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istiod-ca-cert" pod "d2cd527e-0cf0-4513-97b6-7f1f9d8d1584" (UID: "d2cd527e-0cf0-4513-97b6-7f1f9d8d1584"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:02:41.867400 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.867355 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-envoy\") pod \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\" (UID: \"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584\") " Oct 31 20:02:41.867726 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.867702 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-workload-certs" pod "d2cd527e-0cf0-4513-97b6-7f1f9d8d1584" (UID: "d2cd527e-0cf0-4513-97b6-7f1f9d8d1584"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:41.868372 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.868343 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:41.868487 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.868377 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:41.868487 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.868394 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:41.868487 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.868409 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:41.868487 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.868421 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:41.869757 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.869728 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-podinfo" pod "d2cd527e-0cf0-4513-97b6-7f1f9d8d1584" (UID: "d2cd527e-0cf0-4513-97b6-7f1f9d8d1584"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:02:41.870072 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.870050 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-token" pod "d2cd527e-0cf0-4513-97b6-7f1f9d8d1584" (UID: "d2cd527e-0cf0-4513-97b6-7f1f9d8d1584"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:02:41.870166 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.870055 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-envoy" pod "d2cd527e-0cf0-4513-97b6-7f1f9d8d1584" (UID: "d2cd527e-0cf0-4513-97b6-7f1f9d8d1584"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:02:41.870221 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.870204 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-kube-api-access-mbjbx" pod "d2cd527e-0cf0-4513-97b6-7f1f9d8d1584" (UID: "d2cd527e-0cf0-4513-97b6-7f1f9d8d1584"). InnerVolumeSpecName "kube-api-access-mbjbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:02:41.977971 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.970500 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:41.977971 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.970556 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbjbx\" (UniqueName: \"kubernetes.io/projected/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-kube-api-access-mbjbx\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:41.977971 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.970571 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:41.977971 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:41.970586 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:42.678292 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:42.678243 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" event={"ID":"d2cd527e-0cf0-4513-97b6-7f1f9d8d1584","Type":"ContainerDied","Data":"65de04335cec6bc3a1e96da597fc4f1c7126abdbee1737e13420379ae3136b87"} Oct 31 20:02:42.690993 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:42.690865 4910 scope.go:122] "RemoveContainer" containerID="b390e60d26d488396a2d95fc5d1f493fd90143d1a0fe8aa9dc7680bb0d189b5f" Oct 31 20:02:42.715037 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:42.680496 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv" Oct 31 20:02:42.933323 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:02:42.933229 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:02:42.953267 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:42.953232 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv"] Oct 31 20:02:42.962515 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:42.961099 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-qdqkv"] Oct 31 20:02:43.758784 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:43.758750 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2cd527e-0cf0-4513-97b6-7f1f9d8d1584" path="/var/lib/kubelet/pods/d2cd527e-0cf0-4513-97b6-7f1f9d8d1584/volumes" Oct 31 20:02:52.040989 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:52.040949 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-042b89d2-fjctk"] Oct 31 20:02:52.045612 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:52.045581 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-042b89d2-fjctk"] Oct 31 20:02:53.756899 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:53.756857 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e565eb1-e5ec-4638-9412-26692a7cac42" path="/var/lib/kubelet/pods/0e565eb1-e5ec-4638-9412-26692a7cac42/volumes" Oct 31 20:02:55.636859 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:55.636814 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk"] Oct 31 20:02:55.637410 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:55.637080 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk" podUID="27300f49-081d-43d0-b6c5-4d6b55f074ad" containerName="maas-api" containerID="cri-o://d12f6f7db05e4c344e0fd92916175cc490d8966446630f330d0276e2de88147f" gracePeriod=30 Oct 31 20:02:56.357170 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:56.326123 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk" Oct 31 20:02:56.393722 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:56.383672 4910 generic.go:356] "Generic (PLEG): container finished" podID="27300f49-081d-43d0-b6c5-4d6b55f074ad" containerID="d12f6f7db05e4c344e0fd92916175cc490d8966446630f330d0276e2de88147f" exitCode=0 Oct 31 20:02:56.393722 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:56.383758 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk" event={"ID":"27300f49-081d-43d0-b6c5-4d6b55f074ad","Type":"ContainerDied","Data":"d12f6f7db05e4c344e0fd92916175cc490d8966446630f330d0276e2de88147f"} Oct 31 20:02:56.393722 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:56.383819 4910 scope.go:122] "RemoveContainer" containerID="d12f6f7db05e4c344e0fd92916175cc490d8966446630f330d0276e2de88147f" Oct 31 20:02:56.499158 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:56.497900 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27300f49-081d-43d0-b6c5-4d6b55f074ad-kube-api-access-sx4qg\" (UniqueName: \"kubernetes.io/projected/27300f49-081d-43d0-b6c5-4d6b55f074ad-kube-api-access-sx4qg\") pod \"27300f49-081d-43d0-b6c5-4d6b55f074ad\" (UID: \"27300f49-081d-43d0-b6c5-4d6b55f074ad\") " Oct 31 20:02:56.499158 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:56.498005 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27300f49-081d-43d0-b6c5-4d6b55f074ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/27300f49-081d-43d0-b6c5-4d6b55f074ad-metrics-certs\") pod \"27300f49-081d-43d0-b6c5-4d6b55f074ad\" (UID: \"27300f49-081d-43d0-b6c5-4d6b55f074ad\") " Oct 31 20:02:56.499158 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:56.498046 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27300f49-081d-43d0-b6c5-4d6b55f074ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27300f49-081d-43d0-b6c5-4d6b55f074ad-maas-api-tls\") pod \"27300f49-081d-43d0-b6c5-4d6b55f074ad\" (UID: \"27300f49-081d-43d0-b6c5-4d6b55f074ad\") " Oct 31 20:02:56.499158 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:56.498116 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27300f49-081d-43d0-b6c5-4d6b55f074ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27300f49-081d-43d0-b6c5-4d6b55f074ad-openshift-service-ca\") pod \"27300f49-081d-43d0-b6c5-4d6b55f074ad\" (UID: \"27300f49-081d-43d0-b6c5-4d6b55f074ad\") " Oct 31 20:02:56.499512 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:56.499151 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27300f49-081d-43d0-b6c5-4d6b55f074ad-openshift-service-ca" pod "27300f49-081d-43d0-b6c5-4d6b55f074ad" (UID: "27300f49-081d-43d0-b6c5-4d6b55f074ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:02:56.501117 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:56.501087 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27300f49-081d-43d0-b6c5-4d6b55f074ad-metrics-certs" pod "27300f49-081d-43d0-b6c5-4d6b55f074ad" (UID: "27300f49-081d-43d0-b6c5-4d6b55f074ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:02:56.501879 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:56.501849 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27300f49-081d-43d0-b6c5-4d6b55f074ad-maas-api-tls" pod "27300f49-081d-43d0-b6c5-4d6b55f074ad" (UID: "27300f49-081d-43d0-b6c5-4d6b55f074ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:02:56.503376 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:56.503350 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27300f49-081d-43d0-b6c5-4d6b55f074ad-kube-api-access-sx4qg" pod "27300f49-081d-43d0-b6c5-4d6b55f074ad" (UID: "27300f49-081d-43d0-b6c5-4d6b55f074ad"). InnerVolumeSpecName "kube-api-access-sx4qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:02:56.602120 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:56.602078 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27300f49-081d-43d0-b6c5-4d6b55f074ad-maas-api-tls\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:56.602373 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:56.602355 4910 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27300f49-081d-43d0-b6c5-4d6b55f074ad-openshift-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:56.602490 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:56.602477 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx4qg\" (UniqueName: \"kubernetes.io/projected/27300f49-081d-43d0-b6c5-4d6b55f074ad-kube-api-access-sx4qg\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:56.602578 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:56.602567 4910 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27300f49-081d-43d0-b6c5-4d6b55f074ad-metrics-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:02:57.393982 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:57.393939 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk" event={"ID":"27300f49-081d-43d0-b6c5-4d6b55f074ad","Type":"ContainerDied","Data":"17a5a8fc3258694190dbfb6ac66d3117aa59a89d90587b9f233053263fda63e3"} Oct 31 20:02:57.394405 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:57.393952 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk" Oct 31 20:02:57.424797 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:57.424759 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk"] Oct 31 20:02:57.429207 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:57.429180 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-f5c1ab-78fd5996f9-7vhkk"] Oct 31 20:02:57.857932 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:02:57.820783 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27300f49-081d-43d0-b6c5-4d6b55f074ad" path="/var/lib/kubelet/pods/27300f49-081d-43d0-b6c5-4d6b55f074ad/volumes" Oct 31 20:03:02.068089 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:02.068050 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-155ad5a0-8w2wv"] Oct 31 20:03:02.075726 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:02.075688 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-155ad5a0-8w2wv"] Oct 31 20:03:03.769771 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:03.769732 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95eff1aa-9f5f-4009-8dd7-003f1181f586" path="/var/lib/kubelet/pods/95eff1aa-9f5f-4009-8dd7-003f1181f586/volumes" Oct 31 20:03:04.945945 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:04.945650 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4"] Oct 31 20:03:05.507376 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:05.507337 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" podUID="b8398426-b36a-4c2f-b74a-b6676b915a5c" containerName="istio-proxy" containerID="cri-o://5f65a5603972a32ad5d1d1b4af8986e1739153182342d54315a54a49e955cbc4" gracePeriod=30 Oct 31 20:03:10.401779 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.401736 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb"] Oct 31 20:03:10.424158 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.424115 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp"] Oct 31 20:03:10.850228 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.850195 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 20:03:10.913303 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.913272 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-credential-socket\") pod \"b8398426-b36a-4c2f-b74a-b6676b915a5c\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " Oct 31 20:03:10.913504 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.913373 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-workload-certs\") pod \"b8398426-b36a-4c2f-b74a-b6676b915a5c\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " Oct 31 20:03:10.913586 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.913535 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-envoy\") pod \"b8398426-b36a-4c2f-b74a-b6676b915a5c\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " Oct 31 20:03:10.913586 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.913561 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-token\" (UniqueName: \"kubernetes.io/projected/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-token\") pod \"b8398426-b36a-4c2f-b74a-b6676b915a5c\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " Oct 31 20:03:10.913700 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.913594 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8398426-b36a-4c2f-b74a-b6676b915a5c-kube-api-access-xt8x9\" (UniqueName: \"kubernetes.io/projected/b8398426-b36a-4c2f-b74a-b6676b915a5c-kube-api-access-xt8x9\") pod \"b8398426-b36a-4c2f-b74a-b6676b915a5c\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " Oct 31 20:03:10.913700 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.913628 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-credential-socket" pod "b8398426-b36a-4c2f-b74a-b6676b915a5c" (UID: "b8398426-b36a-4c2f-b74a-b6676b915a5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:10.913700 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.913646 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8398426-b36a-4c2f-b74a-b6676b915a5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8398426-b36a-4c2f-b74a-b6676b915a5c-istiod-ca-cert\") pod \"b8398426-b36a-4c2f-b74a-b6676b915a5c\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " Oct 31 20:03:10.913882 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.913715 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-workload-certs" pod "b8398426-b36a-4c2f-b74a-b6676b915a5c" (UID: "b8398426-b36a-4c2f-b74a-b6676b915a5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:10.913882 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.913768 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-podinfo\") pod \"b8398426-b36a-4c2f-b74a-b6676b915a5c\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " Oct 31 20:03:10.913882 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.913812 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-data\") pod \"b8398426-b36a-4c2f-b74a-b6676b915a5c\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " Oct 31 20:03:10.913882 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.913844 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-workload-socket\") pod \"b8398426-b36a-4c2f-b74a-b6676b915a5c\" (UID: \"b8398426-b36a-4c2f-b74a-b6676b915a5c\") " Oct 31 20:03:10.914123 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.914065 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8398426-b36a-4c2f-b74a-b6676b915a5c-istiod-ca-cert" pod "b8398426-b36a-4c2f-b74a-b6676b915a5c" (UID: "b8398426-b36a-4c2f-b74a-b6676b915a5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:03:10.914188 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.914147 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-data" pod "b8398426-b36a-4c2f-b74a-b6676b915a5c" (UID: "b8398426-b36a-4c2f-b74a-b6676b915a5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:10.914393 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.914375 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-workload-socket" pod "b8398426-b36a-4c2f-b74a-b6676b915a5c" (UID: "b8398426-b36a-4c2f-b74a-b6676b915a5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:10.914708 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.914692 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:10.914786 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.914709 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8398426-b36a-4c2f-b74a-b6676b915a5c-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:10.914786 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.914717 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:10.914786 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.914726 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:10.914786 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.914735 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:10.916019 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.915996 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8398426-b36a-4c2f-b74a-b6676b915a5c-kube-api-access-xt8x9" pod "b8398426-b36a-4c2f-b74a-b6676b915a5c" (UID: "b8398426-b36a-4c2f-b74a-b6676b915a5c"). InnerVolumeSpecName "kube-api-access-xt8x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:03:10.916236 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.916207 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-podinfo" pod "b8398426-b36a-4c2f-b74a-b6676b915a5c" (UID: "b8398426-b36a-4c2f-b74a-b6676b915a5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:03:10.916894 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.916875 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-envoy" pod "b8398426-b36a-4c2f-b74a-b6676b915a5c" (UID: "b8398426-b36a-4c2f-b74a-b6676b915a5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:10.917001 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:10.916951 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-token" pod "b8398426-b36a-4c2f-b74a-b6676b915a5c" (UID: "b8398426-b36a-4c2f-b74a-b6676b915a5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:03:11.017005 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:11.016895 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:11.017005 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:11.016948 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:11.017005 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:11.016964 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt8x9\" (UniqueName: \"kubernetes.io/projected/b8398426-b36a-4c2f-b74a-b6676b915a5c-kube-api-access-xt8x9\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:11.017005 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:11.016980 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8398426-b36a-4c2f-b74a-b6676b915a5c-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:11.584320 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:11.584283 4910 generic.go:356] "Generic (PLEG): container finished" podID="b8398426-b36a-4c2f-b74a-b6676b915a5c" containerID="5f65a5603972a32ad5d1d1b4af8986e1739153182342d54315a54a49e955cbc4" exitCode=0 Oct 31 20:03:11.584817 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:11.584333 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" Oct 31 20:03:11.584817 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:11.584370 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" event={"ID":"b8398426-b36a-4c2f-b74a-b6676b915a5c","Type":"ContainerDied","Data":"5f65a5603972a32ad5d1d1b4af8986e1739153182342d54315a54a49e955cbc4"} Oct 31 20:03:11.584817 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:11.584444 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4" event={"ID":"b8398426-b36a-4c2f-b74a-b6676b915a5c","Type":"ContainerDied","Data":"859f6411469f4e47437df05a36db04665149f59e9b383b158faaf932438b8b0e"} Oct 31 20:03:11.584817 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:11.584477 4910 scope.go:122] "RemoveContainer" containerID="5f65a5603972a32ad5d1d1b4af8986e1739153182342d54315a54a49e955cbc4" Oct 31 20:03:11.596840 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:11.596822 4910 scope.go:122] "RemoveContainer" containerID="5f65a5603972a32ad5d1d1b4af8986e1739153182342d54315a54a49e955cbc4" Oct 31 20:03:11.597127 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:03:11.597106 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f65a5603972a32ad5d1d1b4af8986e1739153182342d54315a54a49e955cbc4\": container with ID starting with 5f65a5603972a32ad5d1d1b4af8986e1739153182342d54315a54a49e955cbc4 not found: ID does not exist" containerID="5f65a5603972a32ad5d1d1b4af8986e1739153182342d54315a54a49e955cbc4" Oct 31 20:03:11.597174 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:11.597137 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f65a5603972a32ad5d1d1b4af8986e1739153182342d54315a54a49e955cbc4"} err="failed to get container status \"5f65a5603972a32ad5d1d1b4af8986e1739153182342d54315a54a49e955cbc4\": rpc error: code = NotFound desc = could not find container \"5f65a5603972a32ad5d1d1b4af8986e1739153182342d54315a54a49e955cbc4\": container with ID starting with 5f65a5603972a32ad5d1d1b4af8986e1739153182342d54315a54a49e955cbc4 not found: ID does not exist" Oct 31 20:03:11.612615 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:11.612585 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4"] Oct 31 20:03:11.617269 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:11.617249 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-774z4"] Oct 31 20:03:11.758805 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:11.758766 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8398426-b36a-4c2f-b74a-b6676b915a5c" path="/var/lib/kubelet/pods/b8398426-b36a-4c2f-b74a-b6676b915a5c/volumes" Oct 31 20:03:12.717059 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:03:12.717018 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:03:21.058086 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.058028 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bba4c0c7-z89pz"] Oct 31 20:03:21.067588 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.067556 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bba4c0c7-z89pz"] Oct 31 20:03:21.355985 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.352601 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9d24fb1-f4g8k"] Oct 31 20:03:21.355985 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.353754 4910 cpu_manager.go:414] "containerMap: removing container" podUID="b8398426-b36a-4c2f-b74a-b6676b915a5c" containerName="istio-proxy" Oct 31 20:03:21.355985 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.353777 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8398426-b36a-4c2f-b74a-b6676b915a5c" containerName="istio-proxy" Oct 31 20:03:21.355985 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.353827 4910 cpu_manager.go:414] "containerMap: removing container" podUID="27300f49-081d-43d0-b6c5-4d6b55f074ad" containerName="maas-api" Oct 31 20:03:21.355985 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.353836 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27300f49-081d-43d0-b6c5-4d6b55f074ad" containerName="maas-api" Oct 31 20:03:21.355985 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.353858 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d2cd527e-0cf0-4513-97b6-7f1f9d8d1584" containerName="istio-proxy" Oct 31 20:03:21.355985 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.353865 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2cd527e-0cf0-4513-97b6-7f1f9d8d1584" containerName="istio-proxy" Oct 31 20:03:21.367767 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.367468 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="27300f49-081d-43d0-b6c5-4d6b55f074ad" containerName="maas-api" Oct 31 20:03:21.367767 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.367500 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2cd527e-0cf0-4513-97b6-7f1f9d8d1584" containerName="istio-proxy" Oct 31 20:03:21.367767 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.367526 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8398426-b36a-4c2f-b74a-b6676b915a5c" containerName="istio-proxy" Oct 31 20:03:21.380495 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.373580 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9d24fb1-f4g8k"] Oct 31 20:03:21.380495 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.373695 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d24fb1-f4g8k" Oct 31 20:03:21.385630 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.385607 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 20:03:21.386458 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.386438 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:03:21.501389 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.501354 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb757\" (UniqueName: \"kubernetes.io/projected/6f768640-93b1-4b5a-bbf8-713c315757bb-kube-api-access-tb757\") pod \"maas-api-revoke-sub-d9d24fb1-f4g8k\" (UID: \"6f768640-93b1-4b5a-bbf8-713c315757bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d24fb1-f4g8k" Oct 31 20:03:21.501596 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.501528 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f768640-93b1-4b5a-bbf8-713c315757bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9d24fb1-f4g8k\" (UID: \"6f768640-93b1-4b5a-bbf8-713c315757bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d24fb1-f4g8k" Oct 31 20:03:21.501786 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.501763 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f768640-93b1-4b5a-bbf8-713c315757bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9d24fb1-f4g8k\" (UID: \"6f768640-93b1-4b5a-bbf8-713c315757bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d24fb1-f4g8k" Oct 31 20:03:21.603756 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.603719 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb757\" (UniqueName: \"kubernetes.io/projected/6f768640-93b1-4b5a-bbf8-713c315757bb-kube-api-access-tb757\") pod \"maas-api-revoke-sub-d9d24fb1-f4g8k\" (UID: \"6f768640-93b1-4b5a-bbf8-713c315757bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d24fb1-f4g8k" Oct 31 20:03:21.603964 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.603872 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f768640-93b1-4b5a-bbf8-713c315757bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9d24fb1-f4g8k\" (UID: \"6f768640-93b1-4b5a-bbf8-713c315757bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d24fb1-f4g8k" Oct 31 20:03:21.604116 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.604096 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f768640-93b1-4b5a-bbf8-713c315757bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9d24fb1-f4g8k\" (UID: \"6f768640-93b1-4b5a-bbf8-713c315757bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d24fb1-f4g8k" Oct 31 20:03:21.604847 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.604807 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f768640-93b1-4b5a-bbf8-713c315757bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9d24fb1-f4g8k\" (UID: \"6f768640-93b1-4b5a-bbf8-713c315757bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d24fb1-f4g8k" Oct 31 20:03:21.606787 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.606728 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f768640-93b1-4b5a-bbf8-713c315757bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9d24fb1-f4g8k\" (UID: \"6f768640-93b1-4b5a-bbf8-713c315757bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d24fb1-f4g8k" Oct 31 20:03:21.616842 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.616819 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb757\" (UniqueName: \"kubernetes.io/projected/6f768640-93b1-4b5a-bbf8-713c315757bb-kube-api-access-tb757\") pod \"maas-api-revoke-sub-d9d24fb1-f4g8k\" (UID: \"6f768640-93b1-4b5a-bbf8-713c315757bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d24fb1-f4g8k" Oct 31 20:03:21.695586 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.695554 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d24fb1-f4g8k" Oct 31 20:03:21.773711 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:21.773656 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c21dfb59-b7b0-4dfc-a842-00f203c7b9ce" path="/var/lib/kubelet/pods/c21dfb59-b7b0-4dfc-a842-00f203c7b9ce/volumes" Oct 31 20:03:22.053334 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:03:22.053110 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f768640_93b1_4b5a_bbf8_713c315757bb.slice/crio-b1fb3b7f924a59fe1c6b9a3287f8d2a24d095c58568fc5075864107634b38d22 WatchSource:0}: Error finding container b1fb3b7f924a59fe1c6b9a3287f8d2a24d095c58568fc5075864107634b38d22: Status 404 returned error can't find the container with id b1fb3b7f924a59fe1c6b9a3287f8d2a24d095c58568fc5075864107634b38d22 Oct 31 20:03:22.054026 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:22.053974 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9d24fb1-f4g8k"] Oct 31 20:03:22.878818 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:22.878727 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d24fb1-f4g8k" event={"ID":"6f768640-93b1-4b5a-bbf8-713c315757bb","Type":"ContainerStarted","Data":"95b77623206c8d124c7987acebf617bfb113712d3eb32366c755107068ce9d79"} Oct 31 20:03:22.879260 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:22.878815 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d24fb1-f4g8k" event={"ID":"6f768640-93b1-4b5a-bbf8-713c315757bb","Type":"ContainerStarted","Data":"b1fb3b7f924a59fe1c6b9a3287f8d2a24d095c58568fc5075864107634b38d22"} Oct 31 20:03:22.900853 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:22.900794 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d24fb1-f4g8k" podStartSLOduration=1.9007817249999999 podStartE2EDuration="1.900781725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:03:22.8993683 +0000 UTC m=+2637.262188662" watchObservedRunningTime="2026-10-31 20:03:22.900781725 +0000 UTC m=+2637.263602096" Oct 31 20:03:23.891180 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:23.891142 4910 generic.go:356] "Generic (PLEG): container finished" podID="6f768640-93b1-4b5a-bbf8-713c315757bb" containerID="95b77623206c8d124c7987acebf617bfb113712d3eb32366c755107068ce9d79" exitCode=0 Oct 31 20:03:23.891591 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:23.891196 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d24fb1-f4g8k" event={"ID":"6f768640-93b1-4b5a-bbf8-713c315757bb","Type":"ContainerDied","Data":"95b77623206c8d124c7987acebf617bfb113712d3eb32366c755107068ce9d79"} Oct 31 20:03:25.179241 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:25.179149 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d24fb1-f4g8k" Oct 31 20:03:25.247504 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:25.247457 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f768640-93b1-4b5a-bbf8-713c315757bb-kube-api-access-tb757\" (UniqueName: \"kubernetes.io/projected/6f768640-93b1-4b5a-bbf8-713c315757bb-kube-api-access-tb757\") pod \"6f768640-93b1-4b5a-bbf8-713c315757bb\" (UID: \"6f768640-93b1-4b5a-bbf8-713c315757bb\") " Oct 31 20:03:25.247711 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:25.247621 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f768640-93b1-4b5a-bbf8-713c315757bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f768640-93b1-4b5a-bbf8-713c315757bb-maas-api-xks-ca\") pod \"6f768640-93b1-4b5a-bbf8-713c315757bb\" (UID: \"6f768640-93b1-4b5a-bbf8-713c315757bb\") " Oct 31 20:03:25.247711 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:25.247657 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f768640-93b1-4b5a-bbf8-713c315757bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f768640-93b1-4b5a-bbf8-713c315757bb-maas-api-openshift-ca\") pod \"6f768640-93b1-4b5a-bbf8-713c315757bb\" (UID: \"6f768640-93b1-4b5a-bbf8-713c315757bb\") " Oct 31 20:03:25.248894 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:25.248864 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f768640-93b1-4b5a-bbf8-713c315757bb-maas-api-openshift-ca" pod "6f768640-93b1-4b5a-bbf8-713c315757bb" (UID: "6f768640-93b1-4b5a-bbf8-713c315757bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:03:25.251609 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:25.251580 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f768640-93b1-4b5a-bbf8-713c315757bb-kube-api-access-tb757" pod "6f768640-93b1-4b5a-bbf8-713c315757bb" (UID: "6f768640-93b1-4b5a-bbf8-713c315757bb"). InnerVolumeSpecName "kube-api-access-tb757". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:03:25.251835 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:25.251814 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f768640-93b1-4b5a-bbf8-713c315757bb-maas-api-xks-ca" pod "6f768640-93b1-4b5a-bbf8-713c315757bb" (UID: "6f768640-93b1-4b5a-bbf8-713c315757bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:03:25.350530 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:25.350490 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f768640-93b1-4b5a-bbf8-713c315757bb-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:25.350530 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:25.350526 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f768640-93b1-4b5a-bbf8-713c315757bb-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:25.350760 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:25.350541 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb757\" (UniqueName: \"kubernetes.io/projected/6f768640-93b1-4b5a-bbf8-713c315757bb-kube-api-access-tb757\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:25.427156 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:25.427107 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" containerID="cri-o://de3d92c5bc16ed668458a0627d3246bd098819fed5feef63598db64c9abc785d" gracePeriod=45 Oct 31 20:03:25.462526 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:25.462484 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" containerID="cri-o://1b4aace01da0e0dcf4737a6979ffe08853fa4872303bbbe236aca6871a50c637" gracePeriod=45 Oct 31 20:03:26.055038 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.054952 4910 generic.go:356] "Generic (PLEG): container finished" podID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerID="de3d92c5bc16ed668458a0627d3246bd098819fed5feef63598db64c9abc785d" exitCode=0 Oct 31 20:03:26.059622 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.059568 4910 generic.go:356] "Generic (PLEG): container finished" podID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerID="1b4aace01da0e0dcf4737a6979ffe08853fa4872303bbbe236aca6871a50c637" exitCode=0 Oct 31 20:03:26.065048 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.065023 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d24fb1-f4g8k" Oct 31 20:03:26.070434 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.070392 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" event={"ID":"e7673edf-f90d-4902-a858-c8ec9b5a289b","Type":"ContainerDied","Data":"de3d92c5bc16ed668458a0627d3246bd098819fed5feef63598db64c9abc785d"} Oct 31 20:03:26.070541 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.070458 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" event={"ID":"1a041bbc-a7be-4793-8981-b459959d8ed9","Type":"ContainerDied","Data":"1b4aace01da0e0dcf4737a6979ffe08853fa4872303bbbe236aca6871a50c637"} Oct 31 20:03:26.070541 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.070483 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9d24fb1-f4g8k" event={"ID":"6f768640-93b1-4b5a-bbf8-713c315757bb","Type":"ContainerDied","Data":"b1fb3b7f924a59fe1c6b9a3287f8d2a24d095c58568fc5075864107634b38d22"} Oct 31 20:03:26.070693 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.070545 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1fb3b7f924a59fe1c6b9a3287f8d2a24d095c58568fc5075864107634b38d22" Oct 31 20:03:26.091165 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.087556 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 20:03:26.250026 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.249968 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-tmp-dir\") pod \"e7673edf-f90d-4902-a858-c8ec9b5a289b\" (UID: \"e7673edf-f90d-4902-a858-c8ec9b5a289b\") " Oct 31 20:03:26.254535 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.252098 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 20:03:26.254846 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.254485 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-model-cache\") pod \"e7673edf-f90d-4902-a858-c8ec9b5a289b\" (UID: \"e7673edf-f90d-4902-a858-c8ec9b5a289b\") " Oct 31 20:03:26.254963 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.254943 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-dshm\") pod \"e7673edf-f90d-4902-a858-c8ec9b5a289b\" (UID: \"e7673edf-f90d-4902-a858-c8ec9b5a289b\") " Oct 31 20:03:26.255028 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.254973 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-home\" (UniqueName: \"kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-home\") pod \"e7673edf-f90d-4902-a858-c8ec9b5a289b\" (UID: \"e7673edf-f90d-4902-a858-c8ec9b5a289b\") " Oct 31 20:03:26.255089 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.255037 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7673edf-f90d-4902-a858-c8ec9b5a289b-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7673edf-f90d-4902-a858-c8ec9b5a289b-tls-certs\") pod \"e7673edf-f90d-4902-a858-c8ec9b5a289b\" (UID: \"e7673edf-f90d-4902-a858-c8ec9b5a289b\") " Oct 31 20:03:26.255188 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.255141 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7673edf-f90d-4902-a858-c8ec9b5a289b-kube-api-access-7jwwh\" (UniqueName: \"kubernetes.io/projected/e7673edf-f90d-4902-a858-c8ec9b5a289b-kube-api-access-7jwwh\") pod \"e7673edf-f90d-4902-a858-c8ec9b5a289b\" (UID: \"e7673edf-f90d-4902-a858-c8ec9b5a289b\") " Oct 31 20:03:26.255307 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.255284 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-model-cache" pod "e7673edf-f90d-4902-a858-c8ec9b5a289b" (UID: "e7673edf-f90d-4902-a858-c8ec9b5a289b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:26.255570 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.255555 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-home" pod "e7673edf-f90d-4902-a858-c8ec9b5a289b" (UID: "e7673edf-f90d-4902-a858-c8ec9b5a289b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:26.256053 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.256035 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-model-cache\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:26.256171 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.256160 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-home\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:26.263088 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.263063 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7673edf-f90d-4902-a858-c8ec9b5a289b-tls-certs" pod "e7673edf-f90d-4902-a858-c8ec9b5a289b" (UID: "e7673edf-f90d-4902-a858-c8ec9b5a289b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:03:26.263599 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.263573 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-tmp-dir" pod "e7673edf-f90d-4902-a858-c8ec9b5a289b" (UID: "e7673edf-f90d-4902-a858-c8ec9b5a289b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:26.302416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.302367 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7673edf-f90d-4902-a858-c8ec9b5a289b-kube-api-access-7jwwh" pod "e7673edf-f90d-4902-a858-c8ec9b5a289b" (UID: "e7673edf-f90d-4902-a858-c8ec9b5a289b"). InnerVolumeSpecName "kube-api-access-7jwwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:03:26.307545 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.307483 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-dshm" pod "e7673edf-f90d-4902-a858-c8ec9b5a289b" (UID: "e7673edf-f90d-4902-a858-c8ec9b5a289b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:26.357584 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.357540 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-model-cache\") pod \"1a041bbc-a7be-4793-8981-b459959d8ed9\" (UID: \"1a041bbc-a7be-4793-8981-b459959d8ed9\") " Oct 31 20:03:26.357786 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.357610 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a041bbc-a7be-4793-8981-b459959d8ed9-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a041bbc-a7be-4793-8981-b459959d8ed9-tls-certs\") pod \"1a041bbc-a7be-4793-8981-b459959d8ed9\" (UID: \"1a041bbc-a7be-4793-8981-b459959d8ed9\") " Oct 31 20:03:26.357871 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.357832 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-tmp-dir\") pod \"1a041bbc-a7be-4793-8981-b459959d8ed9\" (UID: \"1a041bbc-a7be-4793-8981-b459959d8ed9\") " Oct 31 20:03:26.357871 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.357842 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-model-cache" pod "1a041bbc-a7be-4793-8981-b459959d8ed9" (UID: "1a041bbc-a7be-4793-8981-b459959d8ed9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:26.358017 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.357872 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-home\" (UniqueName: \"kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-home\") pod \"1a041bbc-a7be-4793-8981-b459959d8ed9\" (UID: \"1a041bbc-a7be-4793-8981-b459959d8ed9\") " Oct 31 20:03:26.358077 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.358023 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-dshm\") pod \"1a041bbc-a7be-4793-8981-b459959d8ed9\" (UID: \"1a041bbc-a7be-4793-8981-b459959d8ed9\") " Oct 31 20:03:26.358077 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.358057 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a041bbc-a7be-4793-8981-b459959d8ed9-kube-api-access-ftvmj\" (UniqueName: \"kubernetes.io/projected/1a041bbc-a7be-4793-8981-b459959d8ed9-kube-api-access-ftvmj\") pod \"1a041bbc-a7be-4793-8981-b459959d8ed9\" (UID: \"1a041bbc-a7be-4793-8981-b459959d8ed9\") " Oct 31 20:03:26.358500 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.358434 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-home" pod "1a041bbc-a7be-4793-8981-b459959d8ed9" (UID: "1a041bbc-a7be-4793-8981-b459959d8ed9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:26.358500 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.358467 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-tmp-dir" pod "1a041bbc-a7be-4793-8981-b459959d8ed9" (UID: "1a041bbc-a7be-4793-8981-b459959d8ed9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:26.359463 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.358788 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-home\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:26.359463 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.358812 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jwwh\" (UniqueName: \"kubernetes.io/projected/e7673edf-f90d-4902-a858-c8ec9b5a289b-kube-api-access-7jwwh\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:26.359463 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.358830 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-tmp-dir\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:26.359463 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.358847 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-model-cache\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:26.359463 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.358862 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7673edf-f90d-4902-a858-c8ec9b5a289b-dshm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:26.359463 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.358876 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7673edf-f90d-4902-a858-c8ec9b5a289b-tls-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:26.359463 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.358889 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-tmp-dir\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:26.360631 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.360599 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a041bbc-a7be-4793-8981-b459959d8ed9-tls-certs" pod "1a041bbc-a7be-4793-8981-b459959d8ed9" (UID: "1a041bbc-a7be-4793-8981-b459959d8ed9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:03:26.361424 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.361397 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-dshm" pod "1a041bbc-a7be-4793-8981-b459959d8ed9" (UID: "1a041bbc-a7be-4793-8981-b459959d8ed9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:26.362341 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.362312 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a041bbc-a7be-4793-8981-b459959d8ed9-kube-api-access-ftvmj" pod "1a041bbc-a7be-4793-8981-b459959d8ed9" (UID: "1a041bbc-a7be-4793-8981-b459959d8ed9"). InnerVolumeSpecName "kube-api-access-ftvmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:03:26.460332 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.460293 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a041bbc-a7be-4793-8981-b459959d8ed9-tls-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:26.460332 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.460331 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a041bbc-a7be-4793-8981-b459959d8ed9-dshm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:26.460557 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.460344 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftvmj\" (UniqueName: \"kubernetes.io/projected/1a041bbc-a7be-4793-8981-b459959d8ed9-kube-api-access-ftvmj\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:26.683242 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.683154 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:03:26.948254 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:26.948144 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 20:03:27.078610 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:27.078576 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" Oct 31 20:03:27.078610 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:27.078597 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb" event={"ID":"e7673edf-f90d-4902-a858-c8ec9b5a289b","Type":"ContainerDied","Data":"0b1a825d7be0dc8b9936243663f2e553d4d1564042a89cc4b631132ebf4a6a0e"} Oct 31 20:03:27.079433 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:27.079410 4910 scope.go:122] "RemoveContainer" containerID="de3d92c5bc16ed668458a0627d3246bd098819fed5feef63598db64c9abc785d" Oct 31 20:03:27.081005 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:27.080606 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" Oct 31 20:03:27.081005 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:27.080654 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp" event={"ID":"1a041bbc-a7be-4793-8981-b459959d8ed9","Type":"ContainerDied","Data":"714a92ed6901d5bb2cd9caa4b0622804552bee2d85d4f2a15cab515579fd9860"} Oct 31 20:03:27.112994 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:27.112968 4910 scope.go:122] "RemoveContainer" containerID="1b4aace01da0e0dcf4737a6979ffe08853fa4872303bbbe236aca6871a50c637" Oct 31 20:03:27.134902 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:27.134873 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb"] Oct 31 20:03:27.174717 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:27.174677 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-23843f/distinct-w0-23843f-kserve-bb7f85f67-wl7fb"] Oct 31 20:03:27.212437 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:27.212345 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp"] Oct 31 20:03:27.216563 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:27.216535 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-23843f/simulator-w0-23843f-kserve-6ffbc975c-mr8hp"] Oct 31 20:03:27.759790 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:27.759754 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" path="/var/lib/kubelet/pods/1a041bbc-a7be-4793-8981-b459959d8ed9/volumes" Oct 31 20:03:27.760403 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:27.760382 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" path="/var/lib/kubelet/pods/e7673edf-f90d-4902-a858-c8ec9b5a289b/volumes" Oct 31 20:03:27.837107 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:27.837059 4910 patch_prober.go:29] interesting pod/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:03:27.837256 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:27.837134 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" podUID="33ed4412-c59e-4ea3-87cd-5d5b485ade85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:03:29.101850 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:29.101813 4910 generic.go:356] "Generic (PLEG): container finished" podID="33ed4412-c59e-4ea3-87cd-5d5b485ade85" containerID="2028ce63f101002960ddc8aa55cb6add8487cf67ee7e06a39880abc0cceba943" exitCode=0 Oct 31 20:03:29.102489 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:29.101885 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" event={"ID":"33ed4412-c59e-4ea3-87cd-5d5b485ade85","Type":"ContainerDied","Data":"2028ce63f101002960ddc8aa55cb6add8487cf67ee7e06a39880abc0cceba943"} Oct 31 20:03:29.102489 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:29.102396 4910 scope.go:122] "RemoveContainer" containerID="2028ce63f101002960ddc8aa55cb6add8487cf67ee7e06a39880abc0cceba943" Oct 31 20:03:29.104454 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:29.104422 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:03:29.104545 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:29.104502 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:03:29.104584 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:29.104547 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:03:29.175892 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:03:29.175849 4910 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f768640_93b1_4b5a_bbf8_713c315757bb.slice\": RecentStats: unable to find data in memory cache]" Oct 31 20:03:30.113082 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:30.113045 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" event={"ID":"33ed4412-c59e-4ea3-87cd-5d5b485ade85","Type":"ContainerStarted","Data":"3801d899a63fa4b29cc8a0564ba1caedb136c8b76385275660900c55baa1d571"} Oct 31 20:03:30.207364 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:30.207327 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:03:30.207547 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:30.207516 4910 patch_prober.go:29] interesting pod/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 31 20:03:30.207593 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:30.207568 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" podUID="33ed4412-c59e-4ea3-87cd-5d5b485ade85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 31 20:03:30.403685 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:03:30.403602 4910 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f768640_93b1_4b5a_bbf8_713c315757bb.slice\": RecentStats: unable to find data in memory cache]" Oct 31 20:03:31.052110 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:31.052063 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-475f069f-76fdd"] Oct 31 20:03:31.059553 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:31.059514 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-475f069f-76fdd"] Oct 31 20:03:31.207490 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:31.207454 4910 patch_prober.go:29] interesting pod/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 31 20:03:31.208023 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:31.207511 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" podUID="33ed4412-c59e-4ea3-87cd-5d5b485ade85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 31 20:03:31.766494 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:31.766462 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de552fee-7ed3-497c-94e3-cc94c6c570a1" path="/var/lib/kubelet/pods/de552fee-7ed3-497c-94e3-cc94c6c570a1/volumes" Oct 31 20:03:32.220417 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:32.218032 4910 patch_prober.go:29] interesting pod/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 31 20:03:32.220417 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:32.218111 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" podUID="33ed4412-c59e-4ea3-87cd-5d5b485ade85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 31 20:03:33.207626 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.207594 4910 patch_prober.go:29] interesting pod/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 31 20:03:33.207934 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.207646 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" podUID="33ed4412-c59e-4ea3-87cd-5d5b485ade85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 31 20:03:33.284304 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.284268 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz"] Oct 31 20:03:33.285000 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.284980 4910 cpu_manager.go:414] "containerMap: removing container" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" Oct 31 20:03:33.285054 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.285003 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" Oct 31 20:03:33.285054 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.285036 4910 cpu_manager.go:414] "containerMap: removing container" podUID="6f768640-93b1-4b5a-bbf8-713c315757bb" containerName="revoke-keys" Oct 31 20:03:33.285054 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.285042 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f768640-93b1-4b5a-bbf8-713c315757bb" containerName="revoke-keys" Oct 31 20:03:33.285054 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.285051 4910 cpu_manager.go:414] "containerMap: removing container" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" Oct 31 20:03:33.285188 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.285055 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" Oct 31 20:03:33.285235 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.285224 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a041bbc-a7be-4793-8981-b459959d8ed9" containerName="main" Oct 31 20:03:33.285271 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.285241 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f768640-93b1-4b5a-bbf8-713c315757bb" containerName="revoke-keys" Oct 31 20:03:33.285271 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.285248 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7673edf-f90d-4902-a858-c8ec9b5a289b" containerName="main" Oct 31 20:03:33.290279 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.290258 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz" Oct 31 20:03:33.296614 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.296593 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:03:33.296798 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.296651 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 20:03:33.304712 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.304689 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz"] Oct 31 20:03:33.414472 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.414435 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9np6\" (UniqueName: \"kubernetes.io/projected/dfc6bdb9-c10a-4503-816f-a99e1d615b6d-kube-api-access-h9np6\") pod \"maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz\" (UID: \"dfc6bdb9-c10a-4503-816f-a99e1d615b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz" Oct 31 20:03:33.414702 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.414628 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc6bdb9-c10a-4503-816f-a99e1d615b6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz\" (UID: \"dfc6bdb9-c10a-4503-816f-a99e1d615b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz" Oct 31 20:03:33.414822 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.414803 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc6bdb9-c10a-4503-816f-a99e1d615b6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz\" (UID: \"dfc6bdb9-c10a-4503-816f-a99e1d615b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz" Oct 31 20:03:33.517041 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.516958 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc6bdb9-c10a-4503-816f-a99e1d615b6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz\" (UID: \"dfc6bdb9-c10a-4503-816f-a99e1d615b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz" Oct 31 20:03:33.517201 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.517119 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9np6\" (UniqueName: \"kubernetes.io/projected/dfc6bdb9-c10a-4503-816f-a99e1d615b6d-kube-api-access-h9np6\") pod \"maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz\" (UID: \"dfc6bdb9-c10a-4503-816f-a99e1d615b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz" Oct 31 20:03:33.517310 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.517293 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc6bdb9-c10a-4503-816f-a99e1d615b6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz\" (UID: \"dfc6bdb9-c10a-4503-816f-a99e1d615b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz" Oct 31 20:03:33.525936 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.517897 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc6bdb9-c10a-4503-816f-a99e1d615b6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz\" (UID: \"dfc6bdb9-c10a-4503-816f-a99e1d615b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz" Oct 31 20:03:33.545943 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.541087 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc6bdb9-c10a-4503-816f-a99e1d615b6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz\" (UID: \"dfc6bdb9-c10a-4503-816f-a99e1d615b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz" Oct 31 20:03:33.556808 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.556775 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9np6\" (UniqueName: \"kubernetes.io/projected/dfc6bdb9-c10a-4503-816f-a99e1d615b6d-kube-api-access-h9np6\") pod \"maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz\" (UID: \"dfc6bdb9-c10a-4503-816f-a99e1d615b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz" Oct 31 20:03:33.611158 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.611120 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz" Oct 31 20:03:33.957837 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:33.956690 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz"] Oct 31 20:03:34.207574 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:34.207476 4910 patch_prober.go:29] interesting pod/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 31 20:03:34.207574 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:34.207550 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" podUID="33ed4412-c59e-4ea3-87cd-5d5b485ade85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 31 20:03:34.274933 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:34.274315 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz" event={"ID":"dfc6bdb9-c10a-4503-816f-a99e1d615b6d","Type":"ContainerStarted","Data":"45afb85cf1a9ba704a694b237d55d25620634783492bf8239ea346d6d4136164"} Oct 31 20:03:35.207492 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:35.207451 4910 patch_prober.go:29] interesting pod/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Oct 31 20:03:35.207885 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:35.207511 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" podUID="33ed4412-c59e-4ea3-87cd-5d5b485ade85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Oct 31 20:03:35.285065 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:35.285027 4910 generic.go:356] "Generic (PLEG): container finished" podID="dfc6bdb9-c10a-4503-816f-a99e1d615b6d" containerID="f724e4a457fafec6ba52165a08bd8006458e529ddc0e320c2291aec3a994eb08" exitCode=0 Oct 31 20:03:35.285258 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:35.285105 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz" event={"ID":"dfc6bdb9-c10a-4503-816f-a99e1d615b6d","Type":"ContainerDied","Data":"f724e4a457fafec6ba52165a08bd8006458e529ddc0e320c2291aec3a994eb08"} Oct 31 20:03:36.212180 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:36.212101 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:03:36.212619 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:36.212605 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:03:36.213450 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:36.213428 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:03:36.521723 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:36.521692 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz" Oct 31 20:03:36.526257 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:36.526224 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfc6bdb9-c10a-4503-816f-a99e1d615b6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc6bdb9-c10a-4503-816f-a99e1d615b6d-maas-api-openshift-ca\") pod \"dfc6bdb9-c10a-4503-816f-a99e1d615b6d\" (UID: \"dfc6bdb9-c10a-4503-816f-a99e1d615b6d\") " Oct 31 20:03:36.526432 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:36.526392 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc6bdb9-c10a-4503-816f-a99e1d615b6d-kube-api-access-h9np6\" (UniqueName: \"kubernetes.io/projected/dfc6bdb9-c10a-4503-816f-a99e1d615b6d-kube-api-access-h9np6\") pod \"dfc6bdb9-c10a-4503-816f-a99e1d615b6d\" (UID: \"dfc6bdb9-c10a-4503-816f-a99e1d615b6d\") " Oct 31 20:03:36.526511 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:36.526492 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfc6bdb9-c10a-4503-816f-a99e1d615b6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc6bdb9-c10a-4503-816f-a99e1d615b6d-maas-api-xks-ca\") pod \"dfc6bdb9-c10a-4503-816f-a99e1d615b6d\" (UID: \"dfc6bdb9-c10a-4503-816f-a99e1d615b6d\") " Oct 31 20:03:36.527008 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:36.526985 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc6bdb9-c10a-4503-816f-a99e1d615b6d-maas-api-openshift-ca" pod "dfc6bdb9-c10a-4503-816f-a99e1d615b6d" (UID: "dfc6bdb9-c10a-4503-816f-a99e1d615b6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:03:36.527294 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:36.527264 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfc6bdb9-c10a-4503-816f-a99e1d615b6d-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:36.529238 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:36.529211 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc6bdb9-c10a-4503-816f-a99e1d615b6d-kube-api-access-h9np6" pod "dfc6bdb9-c10a-4503-816f-a99e1d615b6d" (UID: "dfc6bdb9-c10a-4503-816f-a99e1d615b6d"). InnerVolumeSpecName "kube-api-access-h9np6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:03:36.530085 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:36.530066 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfc6bdb9-c10a-4503-816f-a99e1d615b6d-maas-api-xks-ca" pod "dfc6bdb9-c10a-4503-816f-a99e1d615b6d" (UID: "dfc6bdb9-c10a-4503-816f-a99e1d615b6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:03:36.628651 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:36.628611 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9np6\" (UniqueName: \"kubernetes.io/projected/dfc6bdb9-c10a-4503-816f-a99e1d615b6d-kube-api-access-h9np6\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:36.628651 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:36.628654 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfc6bdb9-c10a-4503-816f-a99e1d615b6d-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:37.306191 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:37.306159 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz" Oct 31 20:03:37.306585 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:37.306188 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz" event={"ID":"dfc6bdb9-c10a-4503-816f-a99e1d615b6d","Type":"ContainerDied","Data":"45afb85cf1a9ba704a694b237d55d25620634783492bf8239ea346d6d4136164"} Oct 31 20:03:37.306585 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:37.306237 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45afb85cf1a9ba704a694b237d55d25620634783492bf8239ea346d6d4136164" Oct 31 20:03:40.488989 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:03:40.488949 4910 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f768640_93b1_4b5a_bbf8_713c315757bb.slice\": RecentStats: unable to find data in memory cache]" Oct 31 20:03:42.718166 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:03:42.718130 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:03:44.107033 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:03:44.106979 4910 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f768640_93b1_4b5a_bbf8_713c315757bb.slice\": RecentStats: unable to find data in memory cache]" Oct 31 20:03:50.575457 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:03:50.575414 4910 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f768640_93b1_4b5a_bbf8_713c315757bb.slice\": RecentStats: unable to find data in memory cache]" Oct 31 20:03:52.897179 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:52.897142 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc"] Oct 31 20:03:52.898412 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:52.898306 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" podUID="94f330fb-bcfb-4a9d-9331-48a619009f36" containerName="istio-proxy" containerID="cri-o://30a881d040a7d7703a5253455c8ce0c06a357f9b6f2ff7f2a026e79515a11069" gracePeriod=30 Oct 31 20:03:52.913261 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:52.912325 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw"] Oct 31 20:03:52.913261 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:52.912627 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" podUID="d1110e52-d926-449c-809b-be04b7d1db94" containerName="istio-proxy" containerID="cri-o://960412caed823b7ab7ba1f5b80dfa9b6a2af2343fa882b8b853457ce1d24b363" gracePeriod=30 Oct 31 20:03:52.927483 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:52.927452 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr"] Oct 31 20:03:52.927715 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:52.927669 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" podUID="687609ff-2575-418e-80e9-c02f7550a362" containerName="istio-proxy" containerID="cri-o://1ede0bf5303e5c793ec3efe82ec21493974a183a3e95567207672227b52116e7" gracePeriod=30 Oct 31 20:03:53.413825 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:53.413792 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx"] Oct 31 20:03:53.414574 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:53.414553 4910 cpu_manager.go:414] "containerMap: removing container" podUID="dfc6bdb9-c10a-4503-816f-a99e1d615b6d" containerName="revoke-keys" Oct 31 20:03:53.414574 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:53.414574 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc6bdb9-c10a-4503-816f-a99e1d615b6d" containerName="revoke-keys" Oct 31 20:03:53.414792 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:53.414777 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc6bdb9-c10a-4503-816f-a99e1d615b6d" containerName="revoke-keys" Oct 31 20:03:53.418607 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:53.418581 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx" Oct 31 20:03:53.428253 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:53.426604 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:03:53.428253 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:53.427306 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 20:03:53.433701 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:53.433676 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx"] Oct 31 20:03:53.473542 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:53.473507 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ef2fa7-0402-472e-85eb-4dbf84ffb317-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx\" (UID: \"45ef2fa7-0402-472e-85eb-4dbf84ffb317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx" Oct 31 20:03:53.473765 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:53.473737 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ef2fa7-0402-472e-85eb-4dbf84ffb317-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx\" (UID: \"45ef2fa7-0402-472e-85eb-4dbf84ffb317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx" Oct 31 20:03:53.473892 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:53.473857 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8bp6\" (UniqueName: \"kubernetes.io/projected/45ef2fa7-0402-472e-85eb-4dbf84ffb317-kube-api-access-g8bp6\") pod \"maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx\" (UID: \"45ef2fa7-0402-472e-85eb-4dbf84ffb317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx" Oct 31 20:03:53.534302 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:53.534268 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:03:53.534499 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:53.534324 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" podUID="94f330fb-bcfb-4a9d-9331-48a619009f36" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:03:53.575753 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:53.575720 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ef2fa7-0402-472e-85eb-4dbf84ffb317-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx\" (UID: \"45ef2fa7-0402-472e-85eb-4dbf84ffb317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx" Oct 31 20:03:53.575955 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:53.575812 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8bp6\" (UniqueName: \"kubernetes.io/projected/45ef2fa7-0402-472e-85eb-4dbf84ffb317-kube-api-access-g8bp6\") pod \"maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx\" (UID: \"45ef2fa7-0402-472e-85eb-4dbf84ffb317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx" Oct 31 20:03:53.575955 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:53.575887 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ef2fa7-0402-472e-85eb-4dbf84ffb317-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx\" (UID: \"45ef2fa7-0402-472e-85eb-4dbf84ffb317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx" Oct 31 20:03:53.576425 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:53.576400 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ef2fa7-0402-472e-85eb-4dbf84ffb317-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx\" (UID: \"45ef2fa7-0402-472e-85eb-4dbf84ffb317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx" Oct 31 20:03:53.578343 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:53.578323 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ef2fa7-0402-472e-85eb-4dbf84ffb317-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx\" (UID: \"45ef2fa7-0402-472e-85eb-4dbf84ffb317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx" Oct 31 20:03:53.591012 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:53.590983 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8bp6\" (UniqueName: \"kubernetes.io/projected/45ef2fa7-0402-472e-85eb-4dbf84ffb317-kube-api-access-g8bp6\") pod \"maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx\" (UID: \"45ef2fa7-0402-472e-85eb-4dbf84ffb317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx" Oct 31 20:03:53.737075 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:53.736986 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx" Oct 31 20:03:53.973014 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:53.972975 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:03:53.973421 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:53.973035 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" podUID="687609ff-2575-418e-80e9-c02f7550a362" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:03:53.996374 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:53.996277 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:03:53.996374 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:53.996330 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" podUID="d1110e52-d926-449c-809b-be04b7d1db94" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:03:54.030734 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:03:54.030700 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ef2fa7_0402_472e_85eb_4dbf84ffb317.slice/crio-ade162b419cc7cd39bdb11fa087d71e32f7df3c368a0e8836e41aada9882bb20 WatchSource:0}: Error finding container ade162b419cc7cd39bdb11fa087d71e32f7df3c368a0e8836e41aada9882bb20: Status 404 returned error can't find the container with id ade162b419cc7cd39bdb11fa087d71e32f7df3c368a0e8836e41aada9882bb20 Oct 31 20:03:54.031618 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:54.031582 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx"] Oct 31 20:03:54.543098 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:54.543060 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx" event={"ID":"45ef2fa7-0402-472e-85eb-4dbf84ffb317","Type":"ContainerStarted","Data":"835c4e4f25410d9b42208ee4f01c378e6d78d9b90e41f6990a925e370a908d81"} Oct 31 20:03:54.543253 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:54.543130 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx" event={"ID":"45ef2fa7-0402-472e-85eb-4dbf84ffb317","Type":"ContainerStarted","Data":"ade162b419cc7cd39bdb11fa087d71e32f7df3c368a0e8836e41aada9882bb20"} Oct 31 20:03:54.562409 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:54.562349 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx" podStartSLOduration=1.562330825 podStartE2EDuration="1.562330825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:03:54.562026091 +0000 UTC m=+2668.924846575" watchObservedRunningTime="2026-10-31 20:03:54.562330825 +0000 UTC m=+2668.925151198" Oct 31 20:03:55.554891 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:55.554803 4910 generic.go:356] "Generic (PLEG): container finished" podID="45ef2fa7-0402-472e-85eb-4dbf84ffb317" containerID="835c4e4f25410d9b42208ee4f01c378e6d78d9b90e41f6990a925e370a908d81" exitCode=0 Oct 31 20:03:55.555325 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:55.554891 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx" event={"ID":"45ef2fa7-0402-472e-85eb-4dbf84ffb317","Type":"ContainerDied","Data":"835c4e4f25410d9b42208ee4f01c378e6d78d9b90e41f6990a925e370a908d81"} Oct 31 20:03:56.836471 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:56.836354 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx" Oct 31 20:03:56.945888 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:56.945848 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45ef2fa7-0402-472e-85eb-4dbf84ffb317-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ef2fa7-0402-472e-85eb-4dbf84ffb317-maas-api-xks-ca\") pod \"45ef2fa7-0402-472e-85eb-4dbf84ffb317\" (UID: \"45ef2fa7-0402-472e-85eb-4dbf84ffb317\") " Oct 31 20:03:56.946113 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:56.945978 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45ef2fa7-0402-472e-85eb-4dbf84ffb317-kube-api-access-g8bp6\" (UniqueName: \"kubernetes.io/projected/45ef2fa7-0402-472e-85eb-4dbf84ffb317-kube-api-access-g8bp6\") pod \"45ef2fa7-0402-472e-85eb-4dbf84ffb317\" (UID: \"45ef2fa7-0402-472e-85eb-4dbf84ffb317\") " Oct 31 20:03:56.946113 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:56.946099 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45ef2fa7-0402-472e-85eb-4dbf84ffb317-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ef2fa7-0402-472e-85eb-4dbf84ffb317-maas-api-openshift-ca\") pod \"45ef2fa7-0402-472e-85eb-4dbf84ffb317\" (UID: \"45ef2fa7-0402-472e-85eb-4dbf84ffb317\") " Oct 31 20:03:56.946532 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:56.946495 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ef2fa7-0402-472e-85eb-4dbf84ffb317-maas-api-openshift-ca" pod "45ef2fa7-0402-472e-85eb-4dbf84ffb317" (UID: "45ef2fa7-0402-472e-85eb-4dbf84ffb317"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:03:56.947034 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:56.947009 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ef2fa7-0402-472e-85eb-4dbf84ffb317-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:56.948746 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:56.948720 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ef2fa7-0402-472e-85eb-4dbf84ffb317-kube-api-access-g8bp6" pod "45ef2fa7-0402-472e-85eb-4dbf84ffb317" (UID: "45ef2fa7-0402-472e-85eb-4dbf84ffb317"). InnerVolumeSpecName "kube-api-access-g8bp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:03:56.948822 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:56.948802 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45ef2fa7-0402-472e-85eb-4dbf84ffb317-maas-api-xks-ca" pod "45ef2fa7-0402-472e-85eb-4dbf84ffb317" (UID: "45ef2fa7-0402-472e-85eb-4dbf84ffb317"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:03:57.048995 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:57.048956 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ef2fa7-0402-472e-85eb-4dbf84ffb317-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:57.048995 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:57.048996 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8bp6\" (UniqueName: \"kubernetes.io/projected/45ef2fa7-0402-472e-85eb-4dbf84ffb317-kube-api-access-g8bp6\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:57.573965 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:57.573898 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx" Oct 31 20:03:57.574204 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:57.573969 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx" event={"ID":"45ef2fa7-0402-472e-85eb-4dbf84ffb317","Type":"ContainerDied","Data":"ade162b419cc7cd39bdb11fa087d71e32f7df3c368a0e8836e41aada9882bb20"} Oct 31 20:03:57.574204 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:57.574065 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ade162b419cc7cd39bdb11fa087d71e32f7df3c368a0e8836e41aada9882bb20" Oct 31 20:03:58.246961 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.245549 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:03:58.308362 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.308328 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:03:58.350731 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.350696 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:03:58.380401 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.375725 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-istio-envoy\") pod \"d1110e52-d926-449c-809b-be04b7d1db94\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " Oct 31 20:03:58.380401 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.376316 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-token\" (UniqueName: \"kubernetes.io/projected/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-token\") pod \"94f330fb-bcfb-4a9d-9331-48a619009f36\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " Oct 31 20:03:58.380401 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.376767 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94f330fb-bcfb-4a9d-9331-48a619009f36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f330fb-bcfb-4a9d-9331-48a619009f36-istiod-ca-cert\") pod \"94f330fb-bcfb-4a9d-9331-48a619009f36\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " Oct 31 20:03:58.380401 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.377253 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f330fb-bcfb-4a9d-9331-48a619009f36-istiod-ca-cert" pod "94f330fb-bcfb-4a9d-9331-48a619009f36" (UID: "94f330fb-bcfb-4a9d-9331-48a619009f36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:03:58.380401 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.377326 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-workload-socket\") pod \"94f330fb-bcfb-4a9d-9331-48a619009f36\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " Oct 31 20:03:58.380401 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.377636 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-workload-socket" pod "94f330fb-bcfb-4a9d-9331-48a619009f36" (UID: "94f330fb-bcfb-4a9d-9331-48a619009f36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:58.380401 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.377706 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-workload-socket\") pod \"d1110e52-d926-449c-809b-be04b7d1db94\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " Oct 31 20:03:58.380401 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.377960 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-workload-socket" pod "d1110e52-d926-449c-809b-be04b7d1db94" (UID: "d1110e52-d926-449c-809b-be04b7d1db94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:58.380401 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.378047 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f330fb-bcfb-4a9d-9331-48a619009f36-kube-api-access-zrsmv\" (UniqueName: \"kubernetes.io/projected/94f330fb-bcfb-4a9d-9331-48a619009f36-kube-api-access-zrsmv\") pod \"94f330fb-bcfb-4a9d-9331-48a619009f36\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " Oct 31 20:03:58.380401 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.378407 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-workload-certs\") pod \"d1110e52-d926-449c-809b-be04b7d1db94\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " Oct 31 20:03:58.380401 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.378702 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-workload-certs" pod "d1110e52-d926-449c-809b-be04b7d1db94" (UID: "d1110e52-d926-449c-809b-be04b7d1db94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:58.380401 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.378778 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-podinfo\") pod \"94f330fb-bcfb-4a9d-9331-48a619009f36\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " Oct 31 20:03:58.380401 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.379214 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-workload-certs\") pod \"94f330fb-bcfb-4a9d-9331-48a619009f36\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " Oct 31 20:03:58.380401 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.379474 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-workload-certs" pod "94f330fb-bcfb-4a9d-9331-48a619009f36" (UID: "94f330fb-bcfb-4a9d-9331-48a619009f36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:58.380401 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.379549 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-istio-data\") pod \"d1110e52-d926-449c-809b-be04b7d1db94\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " Oct 31 20:03:58.380401 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.379771 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-istio-data" pod "d1110e52-d926-449c-809b-be04b7d1db94" (UID: "d1110e52-d926-449c-809b-be04b7d1db94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:58.380401 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.379832 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-credential-socket\") pod \"94f330fb-bcfb-4a9d-9331-48a619009f36\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " Oct 31 20:03:58.392032 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.391989 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-credential-socket" pod "94f330fb-bcfb-4a9d-9331-48a619009f36" (UID: "94f330fb-bcfb-4a9d-9331-48a619009f36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:58.392850 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.392825 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-credential-socket\") pod \"d1110e52-d926-449c-809b-be04b7d1db94\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " Oct 31 20:03:58.392903 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.392872 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1110e52-d926-449c-809b-be04b7d1db94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1110e52-d926-449c-809b-be04b7d1db94-istiod-ca-cert\") pod \"d1110e52-d926-449c-809b-be04b7d1db94\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " Oct 31 20:03:58.392978 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.392960 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1110e52-d926-449c-809b-be04b7d1db94-kube-api-access-qpg4b\" (UniqueName: \"kubernetes.io/projected/d1110e52-d926-449c-809b-be04b7d1db94-kube-api-access-qpg4b\") pod \"d1110e52-d926-449c-809b-be04b7d1db94\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " Oct 31 20:03:58.393100 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.393085 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-envoy\") pod \"94f330fb-bcfb-4a9d-9331-48a619009f36\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " Oct 31 20:03:58.393147 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.393127 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-data\") pod \"94f330fb-bcfb-4a9d-9331-48a619009f36\" (UID: \"94f330fb-bcfb-4a9d-9331-48a619009f36\") " Oct 31 20:03:58.393238 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.393223 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1110e52-d926-449c-809b-be04b7d1db94-istio-token\" (UniqueName: \"kubernetes.io/projected/d1110e52-d926-449c-809b-be04b7d1db94-istio-token\") pod \"d1110e52-d926-449c-809b-be04b7d1db94\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " Oct 31 20:03:58.393341 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.393327 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1110e52-d926-449c-809b-be04b7d1db94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1110e52-d926-449c-809b-be04b7d1db94-istio-podinfo\") pod \"d1110e52-d926-449c-809b-be04b7d1db94\" (UID: \"d1110e52-d926-449c-809b-be04b7d1db94\") " Oct 31 20:03:58.396221 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.395779 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.396221 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.395807 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.396221 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.395822 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.396221 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.395836 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f330fb-bcfb-4a9d-9331-48a619009f36-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.396221 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.395850 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.396221 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.395863 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.396221 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.395876 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.397337 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.396965 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-credential-socket" pod "d1110e52-d926-449c-809b-be04b7d1db94" (UID: "d1110e52-d926-449c-809b-be04b7d1db94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:58.397337 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.397298 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1110e52-d926-449c-809b-be04b7d1db94-istiod-ca-cert" pod "d1110e52-d926-449c-809b-be04b7d1db94" (UID: "d1110e52-d926-449c-809b-be04b7d1db94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:03:58.397853 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.397815 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-data" pod "94f330fb-bcfb-4a9d-9331-48a619009f36" (UID: "94f330fb-bcfb-4a9d-9331-48a619009f36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:58.399079 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.399045 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-podinfo" pod "94f330fb-bcfb-4a9d-9331-48a619009f36" (UID: "94f330fb-bcfb-4a9d-9331-48a619009f36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:03:58.399576 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.399547 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1110e52-d926-449c-809b-be04b7d1db94-kube-api-access-qpg4b" pod "d1110e52-d926-449c-809b-be04b7d1db94" (UID: "d1110e52-d926-449c-809b-be04b7d1db94"). InnerVolumeSpecName "kube-api-access-qpg4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:03:58.400256 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.400226 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f330fb-bcfb-4a9d-9331-48a619009f36-kube-api-access-zrsmv" pod "94f330fb-bcfb-4a9d-9331-48a619009f36" (UID: "94f330fb-bcfb-4a9d-9331-48a619009f36"). InnerVolumeSpecName "kube-api-access-zrsmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:03:58.400449 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.400424 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1110e52-d926-449c-809b-be04b7d1db94-istio-podinfo" pod "d1110e52-d926-449c-809b-be04b7d1db94" (UID: "d1110e52-d926-449c-809b-be04b7d1db94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:03:58.400608 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.400585 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-token" pod "94f330fb-bcfb-4a9d-9331-48a619009f36" (UID: "94f330fb-bcfb-4a9d-9331-48a619009f36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:03:58.401508 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.401476 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1110e52-d926-449c-809b-be04b7d1db94-istio-token" pod "d1110e52-d926-449c-809b-be04b7d1db94" (UID: "d1110e52-d926-449c-809b-be04b7d1db94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:03:58.402678 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.402654 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-envoy" pod "94f330fb-bcfb-4a9d-9331-48a619009f36" (UID: "94f330fb-bcfb-4a9d-9331-48a619009f36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:58.403064 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.403044 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-istio-envoy" pod "d1110e52-d926-449c-809b-be04b7d1db94" (UID: "d1110e52-d926-449c-809b-be04b7d1db94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:58.514995 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.496703 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-credential-socket\") pod \"687609ff-2575-418e-80e9-c02f7550a362\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " Oct 31 20:03:58.514995 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.496864 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/687609ff-2575-418e-80e9-c02f7550a362-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687609ff-2575-418e-80e9-c02f7550a362-istio-podinfo\") pod \"687609ff-2575-418e-80e9-c02f7550a362\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " Oct 31 20:03:58.514995 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.496930 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/687609ff-2575-418e-80e9-c02f7550a362-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687609ff-2575-418e-80e9-c02f7550a362-istiod-ca-cert\") pod \"687609ff-2575-418e-80e9-c02f7550a362\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " Oct 31 20:03:58.514995 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.497038 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-workload-socket\") pod \"687609ff-2575-418e-80e9-c02f7550a362\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " Oct 31 20:03:58.514995 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.497153 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-istio-envoy\") pod \"687609ff-2575-418e-80e9-c02f7550a362\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " Oct 31 20:03:58.514995 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.497216 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-workload-certs\") pod \"687609ff-2575-418e-80e9-c02f7550a362\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " Oct 31 20:03:58.514995 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.497337 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/687609ff-2575-418e-80e9-c02f7550a362-istio-token\" (UniqueName: \"kubernetes.io/projected/687609ff-2575-418e-80e9-c02f7550a362-istio-token\") pod \"687609ff-2575-418e-80e9-c02f7550a362\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " Oct 31 20:03:58.514995 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.497414 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/687609ff-2575-418e-80e9-c02f7550a362-kube-api-access-t5l7w\" (UniqueName: \"kubernetes.io/projected/687609ff-2575-418e-80e9-c02f7550a362-kube-api-access-t5l7w\") pod \"687609ff-2575-418e-80e9-c02f7550a362\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " Oct 31 20:03:58.514995 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.497449 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-istio-data\") pod \"687609ff-2575-418e-80e9-c02f7550a362\" (UID: \"687609ff-2575-418e-80e9-c02f7550a362\") " Oct 31 20:03:58.529686 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.529646 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-workload-socket" pod "687609ff-2575-418e-80e9-c02f7550a362" (UID: "687609ff-2575-418e-80e9-c02f7550a362"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:58.530246 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.530217 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-credential-socket" pod "687609ff-2575-418e-80e9-c02f7550a362" (UID: "687609ff-2575-418e-80e9-c02f7550a362"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:58.533336 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.533235 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/687609ff-2575-418e-80e9-c02f7550a362-istio-podinfo" pod "687609ff-2575-418e-80e9-c02f7550a362" (UID: "687609ff-2575-418e-80e9-c02f7550a362"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:03:58.533647 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.533539 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/687609ff-2575-418e-80e9-c02f7550a362-istiod-ca-cert" pod "687609ff-2575-418e-80e9-c02f7550a362" (UID: "687609ff-2575-418e-80e9-c02f7550a362"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:03:58.534303 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.534044 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-workload-certs" pod "687609ff-2575-418e-80e9-c02f7550a362" (UID: "687609ff-2575-418e-80e9-c02f7550a362"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:58.538364 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.538328 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.538460 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.538365 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1110e52-d926-449c-809b-be04b7d1db94-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.538460 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.538383 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpg4b\" (UniqueName: \"kubernetes.io/projected/d1110e52-d926-449c-809b-be04b7d1db94-kube-api-access-qpg4b\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.538460 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.538399 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687609ff-2575-418e-80e9-c02f7550a362-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.538460 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.538415 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.538460 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.538431 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687609ff-2575-418e-80e9-c02f7550a362-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.538460 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.538447 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.538823 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.538463 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1110e52-d926-449c-809b-be04b7d1db94-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.538823 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.538480 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.538823 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.538496 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1110e52-d926-449c-809b-be04b7d1db94-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.538823 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.538511 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1110e52-d926-449c-809b-be04b7d1db94-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.538823 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.538526 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.538823 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.538540 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.538823 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.538555 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrsmv\" (UniqueName: \"kubernetes.io/projected/94f330fb-bcfb-4a9d-9331-48a619009f36-kube-api-access-zrsmv\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.538823 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.538569 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f330fb-bcfb-4a9d-9331-48a619009f36-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.538823 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.538583 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.541387 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.541212 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-istio-data" pod "687609ff-2575-418e-80e9-c02f7550a362" (UID: "687609ff-2575-418e-80e9-c02f7550a362"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:58.541387 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.541348 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687609ff-2575-418e-80e9-c02f7550a362-istio-token" pod "687609ff-2575-418e-80e9-c02f7550a362" (UID: "687609ff-2575-418e-80e9-c02f7550a362"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:03:58.542040 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.542004 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-istio-envoy" pod "687609ff-2575-418e-80e9-c02f7550a362" (UID: "687609ff-2575-418e-80e9-c02f7550a362"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:03:58.542721 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.542690 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687609ff-2575-418e-80e9-c02f7550a362-kube-api-access-t5l7w" pod "687609ff-2575-418e-80e9-c02f7550a362" (UID: "687609ff-2575-418e-80e9-c02f7550a362"). InnerVolumeSpecName "kube-api-access-t5l7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:03:58.585973 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.585887 4910 generic.go:356] "Generic (PLEG): container finished" podID="687609ff-2575-418e-80e9-c02f7550a362" containerID="1ede0bf5303e5c793ec3efe82ec21493974a183a3e95567207672227b52116e7" exitCode=0 Oct 31 20:03:58.585973 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.585975 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" Oct 31 20:03:58.586229 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.585951 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" event={"ID":"687609ff-2575-418e-80e9-c02f7550a362","Type":"ContainerDied","Data":"1ede0bf5303e5c793ec3efe82ec21493974a183a3e95567207672227b52116e7"} Oct 31 20:03:58.586229 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.586110 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr" event={"ID":"687609ff-2575-418e-80e9-c02f7550a362","Type":"ContainerDied","Data":"b7540d0ea54ef9d441c1292301fdb2a0d6c96530d0335b32ca4ce5f6144fb54b"} Oct 31 20:03:58.586229 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.586163 4910 scope.go:122] "RemoveContainer" containerID="1ede0bf5303e5c793ec3efe82ec21493974a183a3e95567207672227b52116e7" Oct 31 20:03:58.588035 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.587992 4910 generic.go:356] "Generic (PLEG): container finished" podID="94f330fb-bcfb-4a9d-9331-48a619009f36" containerID="30a881d040a7d7703a5253455c8ce0c06a357f9b6f2ff7f2a026e79515a11069" exitCode=0 Oct 31 20:03:58.588248 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.588192 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" event={"ID":"94f330fb-bcfb-4a9d-9331-48a619009f36","Type":"ContainerDied","Data":"30a881d040a7d7703a5253455c8ce0c06a357f9b6f2ff7f2a026e79515a11069"} Oct 31 20:03:58.588354 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.588248 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" event={"ID":"94f330fb-bcfb-4a9d-9331-48a619009f36","Type":"ContainerDied","Data":"6df69d801445fd98cde9f358bfb2d39a847b72671cc76c8d104d0789504cdf8b"} Oct 31 20:03:58.588354 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.588317 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc" Oct 31 20:03:58.591875 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.591845 4910 generic.go:356] "Generic (PLEG): container finished" podID="d1110e52-d926-449c-809b-be04b7d1db94" containerID="960412caed823b7ab7ba1f5b80dfa9b6a2af2343fa882b8b853457ce1d24b363" exitCode=0 Oct 31 20:03:58.592032 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.591950 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" event={"ID":"d1110e52-d926-449c-809b-be04b7d1db94","Type":"ContainerDied","Data":"960412caed823b7ab7ba1f5b80dfa9b6a2af2343fa882b8b853457ce1d24b363"} Oct 31 20:03:58.592032 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.591999 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" event={"ID":"d1110e52-d926-449c-809b-be04b7d1db94","Type":"ContainerDied","Data":"db0e3b1b23f84402429a816fed82f31365d3fbafcb5813ea5d383b0db99defa0"} Oct 31 20:03:58.592165 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.592071 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw" Oct 31 20:03:58.600983 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.600638 4910 scope.go:122] "RemoveContainer" containerID="1ede0bf5303e5c793ec3efe82ec21493974a183a3e95567207672227b52116e7" Oct 31 20:03:58.600983 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:03:58.600940 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ede0bf5303e5c793ec3efe82ec21493974a183a3e95567207672227b52116e7\": container with ID starting with 1ede0bf5303e5c793ec3efe82ec21493974a183a3e95567207672227b52116e7 not found: ID does not exist" containerID="1ede0bf5303e5c793ec3efe82ec21493974a183a3e95567207672227b52116e7" Oct 31 20:03:58.601154 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.600977 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ede0bf5303e5c793ec3efe82ec21493974a183a3e95567207672227b52116e7"} err="failed to get container status \"1ede0bf5303e5c793ec3efe82ec21493974a183a3e95567207672227b52116e7\": rpc error: code = NotFound desc = could not find container \"1ede0bf5303e5c793ec3efe82ec21493974a183a3e95567207672227b52116e7\": container with ID starting with 1ede0bf5303e5c793ec3efe82ec21493974a183a3e95567207672227b52116e7 not found: ID does not exist" Oct 31 20:03:58.601154 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.601000 4910 scope.go:122] "RemoveContainer" containerID="30a881d040a7d7703a5253455c8ce0c06a357f9b6f2ff7f2a026e79515a11069" Oct 31 20:03:58.613323 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.613303 4910 scope.go:122] "RemoveContainer" containerID="30a881d040a7d7703a5253455c8ce0c06a357f9b6f2ff7f2a026e79515a11069" Oct 31 20:03:58.613655 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:03:58.613632 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30a881d040a7d7703a5253455c8ce0c06a357f9b6f2ff7f2a026e79515a11069\": container with ID starting with 30a881d040a7d7703a5253455c8ce0c06a357f9b6f2ff7f2a026e79515a11069 not found: ID does not exist" containerID="30a881d040a7d7703a5253455c8ce0c06a357f9b6f2ff7f2a026e79515a11069" Oct 31 20:03:58.613724 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.613664 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30a881d040a7d7703a5253455c8ce0c06a357f9b6f2ff7f2a026e79515a11069"} err="failed to get container status \"30a881d040a7d7703a5253455c8ce0c06a357f9b6f2ff7f2a026e79515a11069\": rpc error: code = NotFound desc = could not find container \"30a881d040a7d7703a5253455c8ce0c06a357f9b6f2ff7f2a026e79515a11069\": container with ID starting with 30a881d040a7d7703a5253455c8ce0c06a357f9b6f2ff7f2a026e79515a11069 not found: ID does not exist" Oct 31 20:03:58.613724 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.613680 4910 scope.go:122] "RemoveContainer" containerID="960412caed823b7ab7ba1f5b80dfa9b6a2af2343fa882b8b853457ce1d24b363" Oct 31 20:03:58.623041 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.623019 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr"] Oct 31 20:03:58.629120 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.629099 4910 scope.go:122] "RemoveContainer" containerID="e64cde6988e8d049373a6e609b79a0f2e62a4d924b627bb3e04fd1f1f97ee053" Oct 31 20:03:58.630081 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.630057 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-wvnqr"] Oct 31 20:03:58.639339 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.639307 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc"] Oct 31 20:03:58.640279 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.640254 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687609ff-2575-418e-80e9-c02f7550a362-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.640382 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.640282 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5l7w\" (UniqueName: \"kubernetes.io/projected/687609ff-2575-418e-80e9-c02f7550a362-kube-api-access-t5l7w\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.640382 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.640298 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.640382 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.640310 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687609ff-2575-418e-80e9-c02f7550a362-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:03:58.644132 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.644114 4910 scope.go:122] "RemoveContainer" containerID="960412caed823b7ab7ba1f5b80dfa9b6a2af2343fa882b8b853457ce1d24b363" Oct 31 20:03:58.644474 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:03:58.644442 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"960412caed823b7ab7ba1f5b80dfa9b6a2af2343fa882b8b853457ce1d24b363\": container with ID starting with 960412caed823b7ab7ba1f5b80dfa9b6a2af2343fa882b8b853457ce1d24b363 not found: ID does not exist" containerID="960412caed823b7ab7ba1f5b80dfa9b6a2af2343fa882b8b853457ce1d24b363" Oct 31 20:03:58.644572 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.644481 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"960412caed823b7ab7ba1f5b80dfa9b6a2af2343fa882b8b853457ce1d24b363"} err="failed to get container status \"960412caed823b7ab7ba1f5b80dfa9b6a2af2343fa882b8b853457ce1d24b363\": rpc error: code = NotFound desc = could not find container \"960412caed823b7ab7ba1f5b80dfa9b6a2af2343fa882b8b853457ce1d24b363\": container with ID starting with 960412caed823b7ab7ba1f5b80dfa9b6a2af2343fa882b8b853457ce1d24b363 not found: ID does not exist" Oct 31 20:03:58.644572 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.644506 4910 scope.go:122] "RemoveContainer" containerID="e64cde6988e8d049373a6e609b79a0f2e62a4d924b627bb3e04fd1f1f97ee053" Oct 31 20:03:58.644823 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:03:58.644805 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e64cde6988e8d049373a6e609b79a0f2e62a4d924b627bb3e04fd1f1f97ee053\": container with ID starting with e64cde6988e8d049373a6e609b79a0f2e62a4d924b627bb3e04fd1f1f97ee053 not found: ID does not exist" containerID="e64cde6988e8d049373a6e609b79a0f2e62a4d924b627bb3e04fd1f1f97ee053" Oct 31 20:03:58.644877 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.644833 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e64cde6988e8d049373a6e609b79a0f2e62a4d924b627bb3e04fd1f1f97ee053"} err="failed to get container status \"e64cde6988e8d049373a6e609b79a0f2e62a4d924b627bb3e04fd1f1f97ee053\": rpc error: code = NotFound desc = could not find container \"e64cde6988e8d049373a6e609b79a0f2e62a4d924b627bb3e04fd1f1f97ee053\": container with ID starting with e64cde6988e8d049373a6e609b79a0f2e62a4d924b627bb3e04fd1f1f97ee053 not found: ID does not exist" Oct 31 20:03:58.654829 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.654806 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-2zgsc"] Oct 31 20:03:58.672290 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.672262 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw"] Oct 31 20:03:58.676384 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:58.676362 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-tjfkw"] Oct 31 20:03:59.180391 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:03:59.180354 4910 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f768640_93b1_4b5a_bbf8_713c315757bb.slice\": RecentStats: unable to find data in memory cache]" Oct 31 20:03:59.758785 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:59.758745 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="687609ff-2575-418e-80e9-c02f7550a362" path="/var/lib/kubelet/pods/687609ff-2575-418e-80e9-c02f7550a362/volumes" Oct 31 20:03:59.759426 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:59.759407 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94f330fb-bcfb-4a9d-9331-48a619009f36" path="/var/lib/kubelet/pods/94f330fb-bcfb-4a9d-9331-48a619009f36/volumes" Oct 31 20:03:59.760035 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:03:59.760019 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1110e52-d926-449c-809b-be04b7d1db94" path="/var/lib/kubelet/pods/d1110e52-d926-449c-809b-be04b7d1db94/volumes" Oct 31 20:04:00.591667 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:04:00.591633 4910 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f768640_93b1_4b5a_bbf8_713c315757bb.slice\": RecentStats: unable to find data in memory cache]" Oct 31 20:04:00.886532 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:00.885737 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm"] Oct 31 20:04:00.886532 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:00.886079 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" podUID="15884cef-ec21-4fe7-a8b7-f04cf48f4637" containerName="maas-api" containerID="cri-o://dae9ab14c7b29b0ece59205b847ced4e4b4ff1f1115cb998fc6affd2f0f97b55" gracePeriod=30 Oct 31 20:04:01.307735 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:01.307699 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" Oct 31 20:04:01.404956 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:01.390149 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15884cef-ec21-4fe7-a8b7-f04cf48f4637-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15884cef-ec21-4fe7-a8b7-f04cf48f4637-openshift-service-ca\") pod \"15884cef-ec21-4fe7-a8b7-f04cf48f4637\" (UID: \"15884cef-ec21-4fe7-a8b7-f04cf48f4637\") " Oct 31 20:04:01.404956 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:01.390486 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15884cef-ec21-4fe7-a8b7-f04cf48f4637-metrics-certs\" (UniqueName: \"kubernetes.io/secret/15884cef-ec21-4fe7-a8b7-f04cf48f4637-metrics-certs\") pod \"15884cef-ec21-4fe7-a8b7-f04cf48f4637\" (UID: \"15884cef-ec21-4fe7-a8b7-f04cf48f4637\") " Oct 31 20:04:01.404956 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:01.390546 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15884cef-ec21-4fe7-a8b7-f04cf48f4637-kube-api-access-5lk58\" (UniqueName: \"kubernetes.io/projected/15884cef-ec21-4fe7-a8b7-f04cf48f4637-kube-api-access-5lk58\") pod \"15884cef-ec21-4fe7-a8b7-f04cf48f4637\" (UID: \"15884cef-ec21-4fe7-a8b7-f04cf48f4637\") " Oct 31 20:04:01.404956 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:01.390609 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15884cef-ec21-4fe7-a8b7-f04cf48f4637-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15884cef-ec21-4fe7-a8b7-f04cf48f4637-maas-api-tls\") pod \"15884cef-ec21-4fe7-a8b7-f04cf48f4637\" (UID: \"15884cef-ec21-4fe7-a8b7-f04cf48f4637\") " Oct 31 20:04:01.411234 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:01.411171 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15884cef-ec21-4fe7-a8b7-f04cf48f4637-openshift-service-ca" pod "15884cef-ec21-4fe7-a8b7-f04cf48f4637" (UID: "15884cef-ec21-4fe7-a8b7-f04cf48f4637"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:04:01.491155 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:01.485355 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15884cef-ec21-4fe7-a8b7-f04cf48f4637-metrics-certs" pod "15884cef-ec21-4fe7-a8b7-f04cf48f4637" (UID: "15884cef-ec21-4fe7-a8b7-f04cf48f4637"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:04:01.546959 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:01.535129 4910 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15884cef-ec21-4fe7-a8b7-f04cf48f4637-openshift-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.546959 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:01.535167 4910 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15884cef-ec21-4fe7-a8b7-f04cf48f4637-metrics-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.559090 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:01.559045 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15884cef-ec21-4fe7-a8b7-f04cf48f4637-maas-api-tls" pod "15884cef-ec21-4fe7-a8b7-f04cf48f4637" (UID: "15884cef-ec21-4fe7-a8b7-f04cf48f4637"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:04:01.561512 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:01.561483 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15884cef-ec21-4fe7-a8b7-f04cf48f4637-kube-api-access-5lk58" pod "15884cef-ec21-4fe7-a8b7-f04cf48f4637" (UID: "15884cef-ec21-4fe7-a8b7-f04cf48f4637"). InnerVolumeSpecName "kube-api-access-5lk58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:01.650934 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:01.646619 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lk58\" (UniqueName: \"kubernetes.io/projected/15884cef-ec21-4fe7-a8b7-f04cf48f4637-kube-api-access-5lk58\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.650934 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:01.646677 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15884cef-ec21-4fe7-a8b7-f04cf48f4637-maas-api-tls\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:01.754593 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:01.754505 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz"] Oct 31 20:04:01.802353 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:01.801233 4910 generic.go:356] "Generic (PLEG): container finished" podID="15884cef-ec21-4fe7-a8b7-f04cf48f4637" containerID="dae9ab14c7b29b0ece59205b847ced4e4b4ff1f1115cb998fc6affd2f0f97b55" exitCode=0 Oct 31 20:04:01.802353 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:01.801411 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" Oct 31 20:04:01.830331 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:01.830300 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bd1c9f-4nsjz"] Oct 31 20:04:01.830922 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:01.830876 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" event={"ID":"15884cef-ec21-4fe7-a8b7-f04cf48f4637","Type":"ContainerDied","Data":"dae9ab14c7b29b0ece59205b847ced4e4b4ff1f1115cb998fc6affd2f0f97b55"} Oct 31 20:04:01.831083 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:01.831066 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm" event={"ID":"15884cef-ec21-4fe7-a8b7-f04cf48f4637","Type":"ContainerDied","Data":"2d096ee4d1dc42af1f37e0f43695a3af82132fe8c530a2e0717f02589cefbc47"} Oct 31 20:04:01.831213 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:01.831200 4910 scope.go:122] "RemoveContainer" containerID="dae9ab14c7b29b0ece59205b847ced4e4b4ff1f1115cb998fc6affd2f0f97b55" Oct 31 20:04:01.855462 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:01.855434 4910 scope.go:122] "RemoveContainer" containerID="dae9ab14c7b29b0ece59205b847ced4e4b4ff1f1115cb998fc6affd2f0f97b55" Oct 31 20:04:01.855786 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:04:01.855759 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae9ab14c7b29b0ece59205b847ced4e4b4ff1f1115cb998fc6affd2f0f97b55\": container with ID starting with dae9ab14c7b29b0ece59205b847ced4e4b4ff1f1115cb998fc6affd2f0f97b55 not found: ID does not exist" containerID="dae9ab14c7b29b0ece59205b847ced4e4b4ff1f1115cb998fc6affd2f0f97b55" Oct 31 20:04:01.855878 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:01.855796 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae9ab14c7b29b0ece59205b847ced4e4b4ff1f1115cb998fc6affd2f0f97b55"} err="failed to get container status \"dae9ab14c7b29b0ece59205b847ced4e4b4ff1f1115cb998fc6affd2f0f97b55\": rpc error: code = NotFound desc = could not find container \"dae9ab14c7b29b0ece59205b847ced4e4b4ff1f1115cb998fc6affd2f0f97b55\": container with ID starting with dae9ab14c7b29b0ece59205b847ced4e4b4ff1f1115cb998fc6affd2f0f97b55 not found: ID does not exist" Oct 31 20:04:01.864731 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:01.864702 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm"] Oct 31 20:04:01.869791 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:01.869760 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bd1c9f-dbccf485-gmskm"] Oct 31 20:04:03.571499 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:03.571465 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn"] Oct 31 20:04:03.927899 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:03.921192 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15884cef-ec21-4fe7-a8b7-f04cf48f4637" path="/var/lib/kubelet/pods/15884cef-ec21-4fe7-a8b7-f04cf48f4637/volumes" Oct 31 20:04:03.960940 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:03.954325 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfc6bdb9-c10a-4503-816f-a99e1d615b6d" path="/var/lib/kubelet/pods/dfc6bdb9-c10a-4503-816f-a99e1d615b6d/volumes" Oct 31 20:04:04.981179 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:04.981141 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4"] Oct 31 20:04:04.981983 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:04.981928 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" podUID="22f35b72-e954-4d06-ab62-e7a5502a9929" containerName="maas-api" containerID="cri-o://b552b386a0267a1c038217ee045fde6b9e1d1657b712535247f30c40f7604e77" gracePeriod=30 Oct 31 20:04:05.595939 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:05.592972 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx"] Oct 31 20:04:05.664934 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:05.662673 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d7647c-tk6cx"] Oct 31 20:04:05.868963 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:05.863820 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45ef2fa7-0402-472e-85eb-4dbf84ffb317" path="/var/lib/kubelet/pods/45ef2fa7-0402-472e-85eb-4dbf84ffb317/volumes" Oct 31 20:04:05.996221 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:05.992973 4910 generic.go:356] "Generic (PLEG): container finished" podID="22f35b72-e954-4d06-ab62-e7a5502a9929" containerID="b552b386a0267a1c038217ee045fde6b9e1d1657b712535247f30c40f7604e77" exitCode=0 Oct 31 20:04:05.996221 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:05.993138 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" event={"ID":"22f35b72-e954-4d06-ab62-e7a5502a9929","Type":"ContainerDied","Data":"b552b386a0267a1c038217ee045fde6b9e1d1657b712535247f30c40f7604e77"} Oct 31 20:04:06.110854 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:06.110824 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" Oct 31 20:04:06.147777 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:06.147686 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22f35b72-e954-4d06-ab62-e7a5502a9929-metrics-certs\" (UniqueName: \"kubernetes.io/secret/22f35b72-e954-4d06-ab62-e7a5502a9929-metrics-certs\") pod \"22f35b72-e954-4d06-ab62-e7a5502a9929\" (UID: \"22f35b72-e954-4d06-ab62-e7a5502a9929\") " Oct 31 20:04:06.147964 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:06.147810 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f35b72-e954-4d06-ab62-e7a5502a9929-kube-api-access-tgsgn\" (UniqueName: \"kubernetes.io/projected/22f35b72-e954-4d06-ab62-e7a5502a9929-kube-api-access-tgsgn\") pod \"22f35b72-e954-4d06-ab62-e7a5502a9929\" (UID: \"22f35b72-e954-4d06-ab62-e7a5502a9929\") " Oct 31 20:04:06.147964 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:06.147850 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22f35b72-e954-4d06-ab62-e7a5502a9929-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22f35b72-e954-4d06-ab62-e7a5502a9929-maas-api-tls\") pod \"22f35b72-e954-4d06-ab62-e7a5502a9929\" (UID: \"22f35b72-e954-4d06-ab62-e7a5502a9929\") " Oct 31 20:04:06.148055 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:06.147973 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f35b72-e954-4d06-ab62-e7a5502a9929-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22f35b72-e954-4d06-ab62-e7a5502a9929-openshift-service-ca\") pod \"22f35b72-e954-4d06-ab62-e7a5502a9929\" (UID: \"22f35b72-e954-4d06-ab62-e7a5502a9929\") " Oct 31 20:04:06.148391 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:06.148363 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f35b72-e954-4d06-ab62-e7a5502a9929-openshift-service-ca" pod "22f35b72-e954-4d06-ab62-e7a5502a9929" (UID: "22f35b72-e954-4d06-ab62-e7a5502a9929"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:04:06.150385 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:06.150346 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22f35b72-e954-4d06-ab62-e7a5502a9929-maas-api-tls" pod "22f35b72-e954-4d06-ab62-e7a5502a9929" (UID: "22f35b72-e954-4d06-ab62-e7a5502a9929"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:04:06.150521 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:06.150385 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f35b72-e954-4d06-ab62-e7a5502a9929-kube-api-access-tgsgn" pod "22f35b72-e954-4d06-ab62-e7a5502a9929" (UID: "22f35b72-e954-4d06-ab62-e7a5502a9929"). InnerVolumeSpecName "kube-api-access-tgsgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:06.151822 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:06.151795 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22f35b72-e954-4d06-ab62-e7a5502a9929-metrics-certs" pod "22f35b72-e954-4d06-ab62-e7a5502a9929" (UID: "22f35b72-e954-4d06-ab62-e7a5502a9929"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:04:06.250199 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:06.250160 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgsgn\" (UniqueName: \"kubernetes.io/projected/22f35b72-e954-4d06-ab62-e7a5502a9929-kube-api-access-tgsgn\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:06.250199 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:06.250203 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22f35b72-e954-4d06-ab62-e7a5502a9929-maas-api-tls\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:06.250412 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:06.250218 4910 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22f35b72-e954-4d06-ab62-e7a5502a9929-openshift-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:06.250412 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:06.250237 4910 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22f35b72-e954-4d06-ab62-e7a5502a9929-metrics-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:07.053755 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:07.053724 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" Oct 31 20:04:07.054384 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:07.054263 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4" event={"ID":"22f35b72-e954-4d06-ab62-e7a5502a9929","Type":"ContainerDied","Data":"42c8bcd29055c2d0d0a8f0b6679140faf78aa26d494c0708010af25403350548"} Oct 31 20:04:07.054384 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:07.054336 4910 scope.go:122] "RemoveContainer" containerID="b552b386a0267a1c038217ee045fde6b9e1d1657b712535247f30c40f7604e77" Oct 31 20:04:07.085423 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:07.085391 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4"] Oct 31 20:04:07.093407 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:07.093377 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d7647c-58f9ff9b7d-8jnn4"] Oct 31 20:04:07.755535 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:07.755504 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f35b72-e954-4d06-ab62-e7a5502a9929" path="/var/lib/kubelet/pods/22f35b72-e954-4d06-ab62-e7a5502a9929/volumes" Oct 31 20:04:10.098393 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:04:10.098359 4910 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f768640_93b1_4b5a_bbf8_713c315757bb.slice\": RecentStats: unable to find data in memory cache]" Oct 31 20:04:10.098863 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:04:10.098460 4910 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f768640_93b1_4b5a_bbf8_713c315757bb.slice\": RecentStats: unable to find data in memory cache]" Oct 31 20:04:10.649934 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:04:10.649586 4910 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f768640_93b1_4b5a_bbf8_713c315757bb.slice\": RecentStats: unable to find data in memory cache]" Oct 31 20:04:12.540387 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.540348 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8ec13c2-7z45g"] Oct 31 20:04:12.541257 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.541233 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d1110e52-d926-449c-809b-be04b7d1db94" containerName="istio-proxy" Oct 31 20:04:12.541334 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.541262 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1110e52-d926-449c-809b-be04b7d1db94" containerName="istio-proxy" Oct 31 20:04:12.541334 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.541280 4910 cpu_manager.go:414] "containerMap: removing container" podUID="15884cef-ec21-4fe7-a8b7-f04cf48f4637" containerName="maas-api" Oct 31 20:04:12.541334 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.541288 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15884cef-ec21-4fe7-a8b7-f04cf48f4637" containerName="maas-api" Oct 31 20:04:12.541334 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.541308 4910 cpu_manager.go:414] "containerMap: removing container" podUID="687609ff-2575-418e-80e9-c02f7550a362" containerName="istio-proxy" Oct 31 20:04:12.541334 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.541316 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="687609ff-2575-418e-80e9-c02f7550a362" containerName="istio-proxy" Oct 31 20:04:12.541509 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.541341 4910 cpu_manager.go:414] "containerMap: removing container" podUID="22f35b72-e954-4d06-ab62-e7a5502a9929" containerName="maas-api" Oct 31 20:04:12.541509 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.541348 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f35b72-e954-4d06-ab62-e7a5502a9929" containerName="maas-api" Oct 31 20:04:12.541509 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.541385 4910 cpu_manager.go:414] "containerMap: removing container" podUID="94f330fb-bcfb-4a9d-9331-48a619009f36" containerName="istio-proxy" Oct 31 20:04:12.541509 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.541392 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f330fb-bcfb-4a9d-9331-48a619009f36" containerName="istio-proxy" Oct 31 20:04:12.541509 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.541433 4910 cpu_manager.go:414] "containerMap: removing container" podUID="45ef2fa7-0402-472e-85eb-4dbf84ffb317" containerName="revoke-keys" Oct 31 20:04:12.541509 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.541440 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45ef2fa7-0402-472e-85eb-4dbf84ffb317" containerName="revoke-keys" Oct 31 20:04:12.541708 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.541613 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="15884cef-ec21-4fe7-a8b7-f04cf48f4637" containerName="maas-api" Oct 31 20:04:12.541708 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.541628 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="687609ff-2575-418e-80e9-c02f7550a362" containerName="istio-proxy" Oct 31 20:04:12.541708 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.541636 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1110e52-d926-449c-809b-be04b7d1db94" containerName="istio-proxy" Oct 31 20:04:12.541708 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.541642 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f35b72-e954-4d06-ab62-e7a5502a9929" containerName="maas-api" Oct 31 20:04:12.541708 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.541656 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f330fb-bcfb-4a9d-9331-48a619009f36" containerName="istio-proxy" Oct 31 20:04:12.541708 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.541663 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="45ef2fa7-0402-472e-85eb-4dbf84ffb317" containerName="revoke-keys" Oct 31 20:04:12.547403 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.547376 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ec13c2-7z45g" Oct 31 20:04:12.554081 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.554052 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:04:12.554247 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.554151 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 20:04:12.555101 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.555073 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8ec13c2-7z45g"] Oct 31 20:04:12.628738 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.628704 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwnkv\" (UniqueName: \"kubernetes.io/projected/29295413-7152-4a06-8d91-1a7e6819df6c-kube-api-access-gwnkv\") pod \"maas-api-revoke-sub-b8ec13c2-7z45g\" (UID: \"29295413-7152-4a06-8d91-1a7e6819df6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ec13c2-7z45g" Oct 31 20:04:12.628964 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.628746 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29295413-7152-4a06-8d91-1a7e6819df6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8ec13c2-7z45g\" (UID: \"29295413-7152-4a06-8d91-1a7e6819df6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ec13c2-7z45g" Oct 31 20:04:12.628964 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.628937 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29295413-7152-4a06-8d91-1a7e6819df6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8ec13c2-7z45g\" (UID: \"29295413-7152-4a06-8d91-1a7e6819df6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ec13c2-7z45g" Oct 31 20:04:12.716176 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:04:12.716124 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:04:12.730424 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.730386 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwnkv\" (UniqueName: \"kubernetes.io/projected/29295413-7152-4a06-8d91-1a7e6819df6c-kube-api-access-gwnkv\") pod \"maas-api-revoke-sub-b8ec13c2-7z45g\" (UID: \"29295413-7152-4a06-8d91-1a7e6819df6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ec13c2-7z45g" Oct 31 20:04:12.730597 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.730442 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29295413-7152-4a06-8d91-1a7e6819df6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8ec13c2-7z45g\" (UID: \"29295413-7152-4a06-8d91-1a7e6819df6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ec13c2-7z45g" Oct 31 20:04:12.730653 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.730616 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29295413-7152-4a06-8d91-1a7e6819df6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8ec13c2-7z45g\" (UID: \"29295413-7152-4a06-8d91-1a7e6819df6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ec13c2-7z45g" Oct 31 20:04:12.731284 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.731257 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29295413-7152-4a06-8d91-1a7e6819df6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8ec13c2-7z45g\" (UID: \"29295413-7152-4a06-8d91-1a7e6819df6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ec13c2-7z45g" Oct 31 20:04:12.732983 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.732958 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29295413-7152-4a06-8d91-1a7e6819df6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8ec13c2-7z45g\" (UID: \"29295413-7152-4a06-8d91-1a7e6819df6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ec13c2-7z45g" Oct 31 20:04:12.754853 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.754823 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwnkv\" (UniqueName: \"kubernetes.io/projected/29295413-7152-4a06-8d91-1a7e6819df6c-kube-api-access-gwnkv\") pod \"maas-api-revoke-sub-b8ec13c2-7z45g\" (UID: \"29295413-7152-4a06-8d91-1a7e6819df6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ec13c2-7z45g" Oct 31 20:04:12.867348 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:12.867313 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ec13c2-7z45g" Oct 31 20:04:13.204925 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:13.204879 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8ec13c2-7z45g"] Oct 31 20:04:13.206841 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:04:13.206805 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29295413_7152_4a06_8d91_1a7e6819df6c.slice/crio-00aa013351e865cabbcadb2a0b32a47e172556af37f8a44682729268a3405b16 WatchSource:0}: Error finding container 00aa013351e865cabbcadb2a0b32a47e172556af37f8a44682729268a3405b16: Status 404 returned error can't find the container with id 00aa013351e865cabbcadb2a0b32a47e172556af37f8a44682729268a3405b16 Oct 31 20:04:14.108228 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:04:14.108189 4910 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f768640_93b1_4b5a_bbf8_713c315757bb.slice\": RecentStats: unable to find data in memory cache]" Oct 31 20:04:14.166657 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:14.166618 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ec13c2-7z45g" event={"ID":"29295413-7152-4a06-8d91-1a7e6819df6c","Type":"ContainerStarted","Data":"5326e0fc7a07048464248e1cd34399483ac8a2ee2c2aa641761b267262c21112"} Oct 31 20:04:14.166868 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:14.166685 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ec13c2-7z45g" event={"ID":"29295413-7152-4a06-8d91-1a7e6819df6c","Type":"ContainerStarted","Data":"00aa013351e865cabbcadb2a0b32a47e172556af37f8a44682729268a3405b16"} Oct 31 20:04:14.196428 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:14.196361 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ec13c2-7z45g" podStartSLOduration=2.196344087 podStartE2EDuration="2.196344087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:04:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:04:14.188120916 +0000 UTC m=+2688.550941289" watchObservedRunningTime="2026-10-31 20:04:14.196344087 +0000 UTC m=+2688.559164459" Oct 31 20:04:14.598386 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:14.598345 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd"] Oct 31 20:04:14.724537 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:14.724492 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt"] Oct 31 20:04:14.724802 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:14.724767 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" podUID="33ed4412-c59e-4ea3-87cd-5d5b485ade85" containerName="istio-proxy" containerID="cri-o://3801d899a63fa4b29cc8a0564ba1caedb136c8b76385275660900c55baa1d571" gracePeriod=30 Oct 31 20:04:15.190203 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:15.189711 4910 generic.go:356] "Generic (PLEG): container finished" podID="29295413-7152-4a06-8d91-1a7e6819df6c" containerID="5326e0fc7a07048464248e1cd34399483ac8a2ee2c2aa641761b267262c21112" exitCode=0 Oct 31 20:04:15.190203 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:15.189836 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ec13c2-7z45g" event={"ID":"29295413-7152-4a06-8d91-1a7e6819df6c","Type":"ContainerDied","Data":"5326e0fc7a07048464248e1cd34399483ac8a2ee2c2aa641761b267262c21112"} Oct 31 20:04:16.483903 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:16.483835 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ec13c2-7z45g" Oct 31 20:04:16.638675 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:16.638629 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29295413-7152-4a06-8d91-1a7e6819df6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29295413-7152-4a06-8d91-1a7e6819df6c-maas-api-xks-ca\") pod \"29295413-7152-4a06-8d91-1a7e6819df6c\" (UID: \"29295413-7152-4a06-8d91-1a7e6819df6c\") " Oct 31 20:04:16.639023 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:16.638997 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29295413-7152-4a06-8d91-1a7e6819df6c-kube-api-access-gwnkv\" (UniqueName: \"kubernetes.io/projected/29295413-7152-4a06-8d91-1a7e6819df6c-kube-api-access-gwnkv\") pod \"29295413-7152-4a06-8d91-1a7e6819df6c\" (UID: \"29295413-7152-4a06-8d91-1a7e6819df6c\") " Oct 31 20:04:16.639181 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:16.639035 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29295413-7152-4a06-8d91-1a7e6819df6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29295413-7152-4a06-8d91-1a7e6819df6c-maas-api-openshift-ca\") pod \"29295413-7152-4a06-8d91-1a7e6819df6c\" (UID: \"29295413-7152-4a06-8d91-1a7e6819df6c\") " Oct 31 20:04:16.639787 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:16.639756 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29295413-7152-4a06-8d91-1a7e6819df6c-maas-api-openshift-ca" pod "29295413-7152-4a06-8d91-1a7e6819df6c" (UID: "29295413-7152-4a06-8d91-1a7e6819df6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:04:16.640979 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:16.640950 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29295413-7152-4a06-8d91-1a7e6819df6c-maas-api-xks-ca" pod "29295413-7152-4a06-8d91-1a7e6819df6c" (UID: "29295413-7152-4a06-8d91-1a7e6819df6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:04:16.644375 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:16.644352 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29295413-7152-4a06-8d91-1a7e6819df6c-kube-api-access-gwnkv" pod "29295413-7152-4a06-8d91-1a7e6819df6c" (UID: "29295413-7152-4a06-8d91-1a7e6819df6c"). InnerVolumeSpecName "kube-api-access-gwnkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:16.741617 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:16.741519 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwnkv\" (UniqueName: \"kubernetes.io/projected/29295413-7152-4a06-8d91-1a7e6819df6c-kube-api-access-gwnkv\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:16.741617 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:16.741554 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29295413-7152-4a06-8d91-1a7e6819df6c-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:16.741617 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:16.741567 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29295413-7152-4a06-8d91-1a7e6819df6c-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:17.229417 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:17.229384 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ec13c2-7z45g" Oct 31 20:04:17.229637 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:17.229393 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8ec13c2-7z45g" event={"ID":"29295413-7152-4a06-8d91-1a7e6819df6c","Type":"ContainerDied","Data":"00aa013351e865cabbcadb2a0b32a47e172556af37f8a44682729268a3405b16"} Oct 31 20:04:17.229637 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:17.229531 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00aa013351e865cabbcadb2a0b32a47e172556af37f8a44682729268a3405b16" Oct 31 20:04:19.043251 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:19.043204 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-096eeffc-4xmj8"] Oct 31 20:04:19.049831 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:19.049798 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-096eeffc-4xmj8"] Oct 31 20:04:19.194491 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:19.194419 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" podUID="8fa2f1d6-abaa-4c73-8fdc-b91691bfb030" containerName="main" containerID="cri-o://d650d1129e7d54d0c6e988d21653d0da8927eb7aeb36b9f32b8b365e14cd377f" gracePeriod=45 Oct 31 20:04:19.572348 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:19.572321 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 20:04:19.703853 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:19.703768 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-tmp-dir\") pod \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\" (UID: \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\") " Oct 31 20:04:19.704076 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:19.703935 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-home\" (UniqueName: \"kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-home\") pod \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\" (UID: \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\") " Oct 31 20:04:19.704076 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:19.704025 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-tls-certs\") pod \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\" (UID: \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\") " Oct 31 20:04:19.704076 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:19.704055 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-model-cache\") pod \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\" (UID: \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\") " Oct 31 20:04:19.704274 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:19.704108 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-tmp-dir" pod "8fa2f1d6-abaa-4c73-8fdc-b91691bfb030" (UID: "8fa2f1d6-abaa-4c73-8fdc-b91691bfb030"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:19.704274 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:19.704123 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-dshm\") pod \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\" (UID: \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\") " Oct 31 20:04:19.704274 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:19.704241 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-home" pod "8fa2f1d6-abaa-4c73-8fdc-b91691bfb030" (UID: "8fa2f1d6-abaa-4c73-8fdc-b91691bfb030"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:19.704274 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:19.704259 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-kube-api-access-2vkbj\" (UniqueName: \"kubernetes.io/projected/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-kube-api-access-2vkbj\") pod \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\" (UID: \"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030\") " Oct 31 20:04:19.704486 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:19.704390 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-model-cache" pod "8fa2f1d6-abaa-4c73-8fdc-b91691bfb030" (UID: "8fa2f1d6-abaa-4c73-8fdc-b91691bfb030"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:19.705053 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:19.705031 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-tmp-dir\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:19.705053 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:19.705053 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-home\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:19.705201 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:19.705065 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-model-cache\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:19.706489 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:19.706462 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-tls-certs" pod "8fa2f1d6-abaa-4c73-8fdc-b91691bfb030" (UID: "8fa2f1d6-abaa-4c73-8fdc-b91691bfb030"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:04:19.706602 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:19.706530 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-dshm" pod "8fa2f1d6-abaa-4c73-8fdc-b91691bfb030" (UID: "8fa2f1d6-abaa-4c73-8fdc-b91691bfb030"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:19.706705 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:19.706674 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-kube-api-access-2vkbj" pod "8fa2f1d6-abaa-4c73-8fdc-b91691bfb030" (UID: "8fa2f1d6-abaa-4c73-8fdc-b91691bfb030"). InnerVolumeSpecName "kube-api-access-2vkbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:19.759300 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:19.759259 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="850fb91b-2e49-4d05-9e1f-23571e4ca2ec" path="/var/lib/kubelet/pods/850fb91b-2e49-4d05-9e1f-23571e4ca2ec/volumes" Oct 31 20:04:19.806743 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:19.806709 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-tls-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:19.806743 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:19.806741 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-dshm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:19.806974 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:19.806752 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vkbj\" (UniqueName: \"kubernetes.io/projected/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030-kube-api-access-2vkbj\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:20.032627 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.032597 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:04:20.213178 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.213137 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-workload-socket\") pod \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " Oct 31 20:04:20.239003 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.213210 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-podinfo\") pod \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " Oct 31 20:04:20.239003 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.213238 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-data\") pod \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " Oct 31 20:04:20.239003 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.213404 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-credential-socket\") pod \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " Oct 31 20:04:20.239003 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.213507 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-envoy\") pod \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " Oct 31 20:04:20.239003 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.230495 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ed4412-c59e-4ea3-87cd-5d5b485ade85-kube-api-access-tfhzl\" (UniqueName: \"kubernetes.io/projected/33ed4412-c59e-4ea3-87cd-5d5b485ade85-kube-api-access-tfhzl\") pod \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " Oct 31 20:04:20.239003 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.230594 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istiod-ca-cert\") pod \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " Oct 31 20:04:20.239003 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.236430 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-token\" (UniqueName: \"kubernetes.io/projected/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-token\") pod \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " Oct 31 20:04:20.239003 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.236487 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-workload-certs\") pod \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\" (UID: \"33ed4412-c59e-4ea3-87cd-5d5b485ade85\") " Oct 31 20:04:20.239003 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.237180 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-workload-socket" pod "33ed4412-c59e-4ea3-87cd-5d5b485ade85" (UID: "33ed4412-c59e-4ea3-87cd-5d5b485ade85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:20.239003 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.237406 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:20.239003 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.237509 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istiod-ca-cert" pod "33ed4412-c59e-4ea3-87cd-5d5b485ade85" (UID: "33ed4412-c59e-4ea3-87cd-5d5b485ade85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:04:20.239003 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.237623 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-credential-socket" pod "33ed4412-c59e-4ea3-87cd-5d5b485ade85" (UID: "33ed4412-c59e-4ea3-87cd-5d5b485ade85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:20.239003 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.237803 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-data" pod "33ed4412-c59e-4ea3-87cd-5d5b485ade85" (UID: "33ed4412-c59e-4ea3-87cd-5d5b485ade85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:20.239003 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.237974 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-workload-certs" pod "33ed4412-c59e-4ea3-87cd-5d5b485ade85" (UID: "33ed4412-c59e-4ea3-87cd-5d5b485ade85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:20.243145 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.242367 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ed4412-c59e-4ea3-87cd-5d5b485ade85-kube-api-access-tfhzl" pod "33ed4412-c59e-4ea3-87cd-5d5b485ade85" (UID: "33ed4412-c59e-4ea3-87cd-5d5b485ade85"). InnerVolumeSpecName "kube-api-access-tfhzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:20.244327 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.244279 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-envoy" pod "33ed4412-c59e-4ea3-87cd-5d5b485ade85" (UID: "33ed4412-c59e-4ea3-87cd-5d5b485ade85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:20.245447 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.245422 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-token" pod "33ed4412-c59e-4ea3-87cd-5d5b485ade85" (UID: "33ed4412-c59e-4ea3-87cd-5d5b485ade85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:20.252317 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.252281 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-podinfo" pod "33ed4412-c59e-4ea3-87cd-5d5b485ade85" (UID: "33ed4412-c59e-4ea3-87cd-5d5b485ade85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:04:20.279430 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.279361 4910 generic.go:356] "Generic (PLEG): container finished" podID="33ed4412-c59e-4ea3-87cd-5d5b485ade85" containerID="3801d899a63fa4b29cc8a0564ba1caedb136c8b76385275660900c55baa1d571" exitCode=0 Oct 31 20:04:20.279430 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.279419 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" Oct 31 20:04:20.279579 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.279440 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" event={"ID":"33ed4412-c59e-4ea3-87cd-5d5b485ade85","Type":"ContainerDied","Data":"3801d899a63fa4b29cc8a0564ba1caedb136c8b76385275660900c55baa1d571"} Oct 31 20:04:20.279579 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.279519 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt" event={"ID":"33ed4412-c59e-4ea3-87cd-5d5b485ade85","Type":"ContainerDied","Data":"d47686d5f902e9a136a3b6e5d3493fd9dd480765eabc49b0d7e917bf963788d0"} Oct 31 20:04:20.279579 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.279547 4910 scope.go:122] "RemoveContainer" containerID="3801d899a63fa4b29cc8a0564ba1caedb136c8b76385275660900c55baa1d571" Oct 31 20:04:20.281313 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.281285 4910 generic.go:356] "Generic (PLEG): container finished" podID="8fa2f1d6-abaa-4c73-8fdc-b91691bfb030" containerID="d650d1129e7d54d0c6e988d21653d0da8927eb7aeb36b9f32b8b365e14cd377f" exitCode=0 Oct 31 20:04:20.281437 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.281319 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" event={"ID":"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030","Type":"ContainerDied","Data":"d650d1129e7d54d0c6e988d21653d0da8927eb7aeb36b9f32b8b365e14cd377f"} Oct 31 20:04:20.281437 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.281349 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" Oct 31 20:04:20.281437 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.281371 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn" event={"ID":"8fa2f1d6-abaa-4c73-8fdc-b91691bfb030","Type":"ContainerDied","Data":"c8569bdb682689d34525e55d66342a39896dc8c7b5bf20b0f24650d7927fbefb"} Oct 31 20:04:20.306632 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.297760 4910 scope.go:122] "RemoveContainer" containerID="2028ce63f101002960ddc8aa55cb6add8487cf67ee7e06a39880abc0cceba943" Oct 31 20:04:20.315745 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.315719 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn"] Oct 31 20:04:20.319762 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.319736 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-cd9729/sub-test-model-cd9729-kserve-8c4478c65-zw5gn"] Oct 31 20:04:20.320513 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.320494 4910 scope.go:122] "RemoveContainer" containerID="3801d899a63fa4b29cc8a0564ba1caedb136c8b76385275660900c55baa1d571" Oct 31 20:04:20.320865 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:04:20.320840 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3801d899a63fa4b29cc8a0564ba1caedb136c8b76385275660900c55baa1d571\": container with ID starting with 3801d899a63fa4b29cc8a0564ba1caedb136c8b76385275660900c55baa1d571 not found: ID does not exist" containerID="3801d899a63fa4b29cc8a0564ba1caedb136c8b76385275660900c55baa1d571" Oct 31 20:04:20.320958 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.320877 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3801d899a63fa4b29cc8a0564ba1caedb136c8b76385275660900c55baa1d571"} err="failed to get container status \"3801d899a63fa4b29cc8a0564ba1caedb136c8b76385275660900c55baa1d571\": rpc error: code = NotFound desc = could not find container \"3801d899a63fa4b29cc8a0564ba1caedb136c8b76385275660900c55baa1d571\": container with ID starting with 3801d899a63fa4b29cc8a0564ba1caedb136c8b76385275660900c55baa1d571 not found: ID does not exist" Oct 31 20:04:20.320958 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.320899 4910 scope.go:122] "RemoveContainer" containerID="2028ce63f101002960ddc8aa55cb6add8487cf67ee7e06a39880abc0cceba943" Oct 31 20:04:20.321249 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:04:20.321226 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2028ce63f101002960ddc8aa55cb6add8487cf67ee7e06a39880abc0cceba943\": container with ID starting with 2028ce63f101002960ddc8aa55cb6add8487cf67ee7e06a39880abc0cceba943 not found: ID does not exist" containerID="2028ce63f101002960ddc8aa55cb6add8487cf67ee7e06a39880abc0cceba943" Oct 31 20:04:20.321322 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.321256 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2028ce63f101002960ddc8aa55cb6add8487cf67ee7e06a39880abc0cceba943"} err="failed to get container status \"2028ce63f101002960ddc8aa55cb6add8487cf67ee7e06a39880abc0cceba943\": rpc error: code = NotFound desc = could not find container \"2028ce63f101002960ddc8aa55cb6add8487cf67ee7e06a39880abc0cceba943\": container with ID starting with 2028ce63f101002960ddc8aa55cb6add8487cf67ee7e06a39880abc0cceba943 not found: ID does not exist" Oct 31 20:04:20.321322 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.321278 4910 scope.go:122] "RemoveContainer" containerID="d650d1129e7d54d0c6e988d21653d0da8927eb7aeb36b9f32b8b365e14cd377f" Oct 31 20:04:20.333229 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.333213 4910 scope.go:122] "RemoveContainer" containerID="d650d1129e7d54d0c6e988d21653d0da8927eb7aeb36b9f32b8b365e14cd377f" Oct 31 20:04:20.333712 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:04:20.333690 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d650d1129e7d54d0c6e988d21653d0da8927eb7aeb36b9f32b8b365e14cd377f\": container with ID starting with d650d1129e7d54d0c6e988d21653d0da8927eb7aeb36b9f32b8b365e14cd377f not found: ID does not exist" containerID="d650d1129e7d54d0c6e988d21653d0da8927eb7aeb36b9f32b8b365e14cd377f" Oct 31 20:04:20.333823 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.333720 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d650d1129e7d54d0c6e988d21653d0da8927eb7aeb36b9f32b8b365e14cd377f"} err="failed to get container status \"d650d1129e7d54d0c6e988d21653d0da8927eb7aeb36b9f32b8b365e14cd377f\": rpc error: code = NotFound desc = could not find container \"d650d1129e7d54d0c6e988d21653d0da8927eb7aeb36b9f32b8b365e14cd377f\": container with ID starting with d650d1129e7d54d0c6e988d21653d0da8927eb7aeb36b9f32b8b365e14cd377f not found: ID does not exist" Oct 31 20:04:20.334453 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.334431 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt"] Oct 31 20:04:20.339095 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.339073 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d7647c-openshift-default-5745b7cd5f-xw6gt"] Oct 31 20:04:20.340519 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.340495 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:20.340629 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.340532 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:20.340629 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.340565 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:20.340629 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.340582 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:20.340629 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.340597 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfhzl\" (UniqueName: \"kubernetes.io/projected/33ed4412-c59e-4ea3-87cd-5d5b485ade85-kube-api-access-tfhzl\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:20.340824 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.340635 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:20.340824 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.340651 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33ed4412-c59e-4ea3-87cd-5d5b485ade85-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:20.340824 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:20.340665 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33ed4412-c59e-4ea3-87cd-5d5b485ade85-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:20.705880 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:04:20.705846 4910 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f768640_93b1_4b5a_bbf8_713c315757bb.slice\": RecentStats: unable to find data in memory cache]" Oct 31 20:04:21.755800 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:21.755766 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33ed4412-c59e-4ea3-87cd-5d5b485ade85" path="/var/lib/kubelet/pods/33ed4412-c59e-4ea3-87cd-5d5b485ade85/volumes" Oct 31 20:04:21.756377 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:21.756360 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fa2f1d6-abaa-4c73-8fdc-b91691bfb030" path="/var/lib/kubelet/pods/8fa2f1d6-abaa-4c73-8fdc-b91691bfb030/volumes" Oct 31 20:04:22.766718 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.766682 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b37081-57vcz"] Oct 31 20:04:22.767346 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.767330 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d1110e52-d926-449c-809b-be04b7d1db94" containerName="istio-proxy" Oct 31 20:04:22.767394 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.767347 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1110e52-d926-449c-809b-be04b7d1db94" containerName="istio-proxy" Oct 31 20:04:22.767394 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.767378 4910 cpu_manager.go:414] "containerMap: removing container" podUID="8fa2f1d6-abaa-4c73-8fdc-b91691bfb030" containerName="main" Oct 31 20:04:22.767394 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.767384 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa2f1d6-abaa-4c73-8fdc-b91691bfb030" containerName="main" Oct 31 20:04:22.767394 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.767392 4910 cpu_manager.go:414] "containerMap: removing container" podUID="33ed4412-c59e-4ea3-87cd-5d5b485ade85" containerName="istio-proxy" Oct 31 20:04:22.767530 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.767397 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33ed4412-c59e-4ea3-87cd-5d5b485ade85" containerName="istio-proxy" Oct 31 20:04:22.767530 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.767417 4910 cpu_manager.go:414] "containerMap: removing container" podUID="29295413-7152-4a06-8d91-1a7e6819df6c" containerName="revoke-keys" Oct 31 20:04:22.767530 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.767422 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29295413-7152-4a06-8d91-1a7e6819df6c" containerName="revoke-keys" Oct 31 20:04:22.767530 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.767433 4910 cpu_manager.go:414] "containerMap: removing container" podUID="33ed4412-c59e-4ea3-87cd-5d5b485ade85" containerName="istio-proxy" Oct 31 20:04:22.767530 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.767438 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33ed4412-c59e-4ea3-87cd-5d5b485ade85" containerName="istio-proxy" Oct 31 20:04:22.767696 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.767540 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="29295413-7152-4a06-8d91-1a7e6819df6c" containerName="revoke-keys" Oct 31 20:04:22.767696 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.767551 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="33ed4412-c59e-4ea3-87cd-5d5b485ade85" containerName="istio-proxy" Oct 31 20:04:22.767696 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.767566 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1110e52-d926-449c-809b-be04b7d1db94" containerName="istio-proxy" Oct 31 20:04:22.767696 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.767576 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa2f1d6-abaa-4c73-8fdc-b91691bfb030" containerName="main" Oct 31 20:04:22.767696 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.767583 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="33ed4412-c59e-4ea3-87cd-5d5b485ade85" containerName="istio-proxy" Oct 31 20:04:22.771103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.771083 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b37081-57vcz" Oct 31 20:04:22.777265 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.777237 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:04:22.777411 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.777353 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 20:04:22.781376 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.781357 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b37081-57vcz"] Oct 31 20:04:22.819174 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.819139 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4936de05-9195-4a0a-b44c-75049422759b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2b37081-57vcz\" (UID: \"4936de05-9195-4a0a-b44c-75049422759b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b37081-57vcz" Oct 31 20:04:22.819531 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.819504 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk2x4\" (UniqueName: \"kubernetes.io/projected/4936de05-9195-4a0a-b44c-75049422759b-kube-api-access-fk2x4\") pod \"maas-api-revoke-sub-a2b37081-57vcz\" (UID: \"4936de05-9195-4a0a-b44c-75049422759b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b37081-57vcz" Oct 31 20:04:22.819736 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.819707 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4936de05-9195-4a0a-b44c-75049422759b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2b37081-57vcz\" (UID: \"4936de05-9195-4a0a-b44c-75049422759b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b37081-57vcz" Oct 31 20:04:22.921651 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.921610 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4936de05-9195-4a0a-b44c-75049422759b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2b37081-57vcz\" (UID: \"4936de05-9195-4a0a-b44c-75049422759b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b37081-57vcz" Oct 31 20:04:22.921881 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.921706 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk2x4\" (UniqueName: \"kubernetes.io/projected/4936de05-9195-4a0a-b44c-75049422759b-kube-api-access-fk2x4\") pod \"maas-api-revoke-sub-a2b37081-57vcz\" (UID: \"4936de05-9195-4a0a-b44c-75049422759b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b37081-57vcz" Oct 31 20:04:22.921881 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.921777 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4936de05-9195-4a0a-b44c-75049422759b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2b37081-57vcz\" (UID: \"4936de05-9195-4a0a-b44c-75049422759b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b37081-57vcz" Oct 31 20:04:22.922452 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.922420 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4936de05-9195-4a0a-b44c-75049422759b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2b37081-57vcz\" (UID: \"4936de05-9195-4a0a-b44c-75049422759b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b37081-57vcz" Oct 31 20:04:22.924378 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.924351 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4936de05-9195-4a0a-b44c-75049422759b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2b37081-57vcz\" (UID: \"4936de05-9195-4a0a-b44c-75049422759b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b37081-57vcz" Oct 31 20:04:22.936721 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:22.936693 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk2x4\" (UniqueName: \"kubernetes.io/projected/4936de05-9195-4a0a-b44c-75049422759b-kube-api-access-fk2x4\") pod \"maas-api-revoke-sub-a2b37081-57vcz\" (UID: \"4936de05-9195-4a0a-b44c-75049422759b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b37081-57vcz" Oct 31 20:04:23.088130 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:23.088094 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b37081-57vcz" Oct 31 20:04:23.377031 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:23.377003 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b37081-57vcz"] Oct 31 20:04:23.377506 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:04:23.377478 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4936de05_9195_4a0a_b44c_75049422759b.slice/crio-a0715018f576cf4e4aabc3ab8523e9ce8a6913e393f7a12f03a1d0cce1975c41 WatchSource:0}: Error finding container a0715018f576cf4e4aabc3ab8523e9ce8a6913e393f7a12f03a1d0cce1975c41: Status 404 returned error can't find the container with id a0715018f576cf4e4aabc3ab8523e9ce8a6913e393f7a12f03a1d0cce1975c41 Oct 31 20:04:24.351011 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.350973 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b37081-57vcz" event={"ID":"4936de05-9195-4a0a-b44c-75049422759b","Type":"ContainerStarted","Data":"7748b672b8bcf3bc88c4d53b7a68dbce8ae7ea08910751ab8e2992e7089db3b5"} Oct 31 20:04:24.351367 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.351081 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b37081-57vcz" event={"ID":"4936de05-9195-4a0a-b44c-75049422759b","Type":"ContainerStarted","Data":"a0715018f576cf4e4aabc3ab8523e9ce8a6913e393f7a12f03a1d0cce1975c41"} Oct 31 20:04:24.798764 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.798688 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf"] Oct 31 20:04:24.803165 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.803147 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.821774 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.821748 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf"] Oct 31 20:04:24.856634 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.856517 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istiod-ca-cert\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.856634 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.856577 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-envoy\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.856923 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.856695 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-workload-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.856923 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.856860 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlsph\" (UniqueName: \"kubernetes.io/projected/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-kube-api-access-mlsph\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.857072 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.856955 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-data\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.857072 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.857023 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-credential-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.857231 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.857124 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-podinfo\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.857289 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.857231 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-token\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.857289 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.857260 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-workload-certs\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.958774 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.958730 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-token\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.958774 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.958776 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-workload-certs\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.959059 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.958832 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istiod-ca-cert\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.959059 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.958857 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-envoy\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.959059 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.958885 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-workload-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.959059 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.958979 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlsph\" (UniqueName: \"kubernetes.io/projected/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-kube-api-access-mlsph\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.959059 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.959050 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-data\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.959794 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.959105 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-credential-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.959794 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.959204 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-podinfo\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.959794 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.959370 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-workload-certs\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.959794 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.959741 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-credential-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.960080 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.959987 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-data\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.960080 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.960014 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-workload-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.960666 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.960638 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istiod-ca-cert\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.962090 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.962063 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-envoy\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.962498 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.962477 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-podinfo\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.971479 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.971449 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-token\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:24.971597 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:24.971521 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlsph\" (UniqueName: \"kubernetes.io/projected/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-kube-api-access-mlsph\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:25.117197 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:25.117163 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:25.362072 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:25.362037 4910 generic.go:356] "Generic (PLEG): container finished" podID="4936de05-9195-4a0a-b44c-75049422759b" containerID="7748b672b8bcf3bc88c4d53b7a68dbce8ae7ea08910751ab8e2992e7089db3b5" exitCode=0 Oct 31 20:04:25.362608 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:25.362123 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b37081-57vcz" event={"ID":"4936de05-9195-4a0a-b44c-75049422759b","Type":"ContainerDied","Data":"7748b672b8bcf3bc88c4d53b7a68dbce8ae7ea08910751ab8e2992e7089db3b5"} Oct 31 20:04:25.431359 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:25.429947 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf"] Oct 31 20:04:25.456078 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:25.455629 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:04:25.456078 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:25.455724 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:04:25.456078 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:25.455766 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:04:25.585731 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:25.585700 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b37081-57vcz" Oct 31 20:04:25.706991 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:25.698328 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4936de05-9195-4a0a-b44c-75049422759b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4936de05-9195-4a0a-b44c-75049422759b-maas-api-xks-ca\") pod \"4936de05-9195-4a0a-b44c-75049422759b\" (UID: \"4936de05-9195-4a0a-b44c-75049422759b\") " Oct 31 20:04:25.706991 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:25.698413 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4936de05-9195-4a0a-b44c-75049422759b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4936de05-9195-4a0a-b44c-75049422759b-maas-api-openshift-ca\") pod \"4936de05-9195-4a0a-b44c-75049422759b\" (UID: \"4936de05-9195-4a0a-b44c-75049422759b\") " Oct 31 20:04:25.706991 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:25.698617 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4936de05-9195-4a0a-b44c-75049422759b-kube-api-access-fk2x4\" (UniqueName: \"kubernetes.io/projected/4936de05-9195-4a0a-b44c-75049422759b-kube-api-access-fk2x4\") pod \"4936de05-9195-4a0a-b44c-75049422759b\" (UID: \"4936de05-9195-4a0a-b44c-75049422759b\") " Oct 31 20:04:25.733933 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:25.733228 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4936de05-9195-4a0a-b44c-75049422759b-maas-api-openshift-ca" pod "4936de05-9195-4a0a-b44c-75049422759b" (UID: "4936de05-9195-4a0a-b44c-75049422759b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:04:25.807563 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:25.806279 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4936de05-9195-4a0a-b44c-75049422759b-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:25.915934 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:25.908878 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4936de05-9195-4a0a-b44c-75049422759b-maas-api-xks-ca" pod "4936de05-9195-4a0a-b44c-75049422759b" (UID: "4936de05-9195-4a0a-b44c-75049422759b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:04:25.915934 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:25.912795 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4936de05-9195-4a0a-b44c-75049422759b-kube-api-access-fk2x4" pod "4936de05-9195-4a0a-b44c-75049422759b" (UID: "4936de05-9195-4a0a-b44c-75049422759b"). InnerVolumeSpecName "kube-api-access-fk2x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:25.915934 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:25.913180 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4936de05-9195-4a0a-b44c-75049422759b-kube-api-access-fk2x4\" (UniqueName: \"kubernetes.io/projected/4936de05-9195-4a0a-b44c-75049422759b-kube-api-access-fk2x4\") pod \"4936de05-9195-4a0a-b44c-75049422759b\" (UID: \"4936de05-9195-4a0a-b44c-75049422759b\") " Oct 31 20:04:25.915934 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:25.914393 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4936de05-9195-4a0a-b44c-75049422759b-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:25.915934 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:04:25.914488 4910 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/4936de05-9195-4a0a-b44c-75049422759b/volumes/kubernetes.io~projected/kube-api-access-fk2x4 Oct 31 20:04:25.915934 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:25.914529 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4936de05-9195-4a0a-b44c-75049422759b-kube-api-access-fk2x4" pod "4936de05-9195-4a0a-b44c-75049422759b" (UID: "4936de05-9195-4a0a-b44c-75049422759b"). InnerVolumeSpecName "kube-api-access-fk2x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:26.016360 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:26.016330 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk2x4\" (UniqueName: \"kubernetes.io/projected/4936de05-9195-4a0a-b44c-75049422759b-kube-api-access-fk2x4\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:26.394128 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:26.394091 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b37081-57vcz" event={"ID":"4936de05-9195-4a0a-b44c-75049422759b","Type":"ContainerDied","Data":"a0715018f576cf4e4aabc3ab8523e9ce8a6913e393f7a12f03a1d0cce1975c41"} Oct 31 20:04:26.394728 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:26.394705 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0715018f576cf4e4aabc3ab8523e9ce8a6913e393f7a12f03a1d0cce1975c41" Oct 31 20:04:26.394857 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:26.394138 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b37081-57vcz" Oct 31 20:04:26.396891 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:26.396861 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" event={"ID":"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13","Type":"ContainerStarted","Data":"377cb0825ffe594216103119e3715f89a5b6cd0405aaba7c79e71a297479a351"} Oct 31 20:04:26.397002 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:26.396930 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" event={"ID":"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13","Type":"ContainerStarted","Data":"839b548b30793416af7eb70514ecf5cde97864a8e86320a24622a7b25dede4a6"} Oct 31 20:04:26.646007 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:26.645824 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" podStartSLOduration=2.645802425 podStartE2EDuration="2.645802425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:04:26.425570962 +0000 UTC m=+2700.788391334" watchObservedRunningTime="2026-10-31 20:04:26.645802425 +0000 UTC m=+2701.008622800" Oct 31 20:04:26.857889 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:26.857852 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 20:04:26.885658 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:26.885630 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 20:04:27.121937 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:27.117972 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:27.141932 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:27.130047 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body= Oct 31 20:04:27.141932 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:27.130115 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" podUID="59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" Oct 31 20:04:28.117551 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:28.117516 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body= Oct 31 20:04:28.117992 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:28.117573 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" podUID="59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" Oct 31 20:04:29.117495 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:29.117458 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body= Oct 31 20:04:29.117730 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:29.117515 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" podUID="59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" Oct 31 20:04:29.632933 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:29.630319 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" podUID="c20b518f-fe19-47b6-88a6-aec10c91cf71" containerName="main" containerID="cri-o://e21998d7c0ef6baefefd073f3bcf21172c6058f0b104c9e22f806771ff24c0f7" gracePeriod=45 Oct 31 20:04:29.703957 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:29.703901 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" podUID="c20b518f-fe19-47b6-88a6-aec10c91cf71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.244:8000/ready\": dial tcp 10.132.0.244:8000: connect: connection refused" Oct 31 20:04:30.059041 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.058593 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 20:04:30.127938 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.118184 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body= Oct 31 20:04:30.146072 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.146027 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" podUID="59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" Oct 31 20:04:30.147272 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.147244 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-dshm\") pod \"c20b518f-fe19-47b6-88a6-aec10c91cf71\" (UID: \"c20b518f-fe19-47b6-88a6-aec10c91cf71\") " Oct 31 20:04:30.147840 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.147819 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-home\" (UniqueName: \"kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-home\") pod \"c20b518f-fe19-47b6-88a6-aec10c91cf71\" (UID: \"c20b518f-fe19-47b6-88a6-aec10c91cf71\") " Oct 31 20:04:30.148330 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.148205 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-home" pod "c20b518f-fe19-47b6-88a6-aec10c91cf71" (UID: "c20b518f-fe19-47b6-88a6-aec10c91cf71"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:30.148596 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.148577 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c20b518f-fe19-47b6-88a6-aec10c91cf71-kube-api-access-8crgx\" (UniqueName: \"kubernetes.io/projected/c20b518f-fe19-47b6-88a6-aec10c91cf71-kube-api-access-8crgx\") pod \"c20b518f-fe19-47b6-88a6-aec10c91cf71\" (UID: \"c20b518f-fe19-47b6-88a6-aec10c91cf71\") " Oct 31 20:04:30.149158 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.149139 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c20b518f-fe19-47b6-88a6-aec10c91cf71-tls-certs\" (UniqueName: \"kubernetes.io/secret/c20b518f-fe19-47b6-88a6-aec10c91cf71-tls-certs\") pod \"c20b518f-fe19-47b6-88a6-aec10c91cf71\" (UID: \"c20b518f-fe19-47b6-88a6-aec10c91cf71\") " Oct 31 20:04:30.149462 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.149446 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-tmp-dir\") pod \"c20b518f-fe19-47b6-88a6-aec10c91cf71\" (UID: \"c20b518f-fe19-47b6-88a6-aec10c91cf71\") " Oct 31 20:04:30.149952 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.149819 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-tmp-dir" pod "c20b518f-fe19-47b6-88a6-aec10c91cf71" (UID: "c20b518f-fe19-47b6-88a6-aec10c91cf71"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:30.150039 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.150016 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-model-cache\") pod \"c20b518f-fe19-47b6-88a6-aec10c91cf71\" (UID: \"c20b518f-fe19-47b6-88a6-aec10c91cf71\") " Oct 31 20:04:30.150398 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.150369 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-model-cache" pod "c20b518f-fe19-47b6-88a6-aec10c91cf71" (UID: "c20b518f-fe19-47b6-88a6-aec10c91cf71"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:30.150871 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.150845 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-tmp-dir\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:30.150996 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.150873 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-model-cache\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:30.150996 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.150898 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-home\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:30.151262 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.151230 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-dshm" pod "c20b518f-fe19-47b6-88a6-aec10c91cf71" (UID: "c20b518f-fe19-47b6-88a6-aec10c91cf71"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:30.152311 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.152276 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c20b518f-fe19-47b6-88a6-aec10c91cf71-kube-api-access-8crgx" pod "c20b518f-fe19-47b6-88a6-aec10c91cf71" (UID: "c20b518f-fe19-47b6-88a6-aec10c91cf71"). InnerVolumeSpecName "kube-api-access-8crgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:30.152676 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.152653 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c20b518f-fe19-47b6-88a6-aec10c91cf71-tls-certs" pod "c20b518f-fe19-47b6-88a6-aec10c91cf71" (UID: "c20b518f-fe19-47b6-88a6-aec10c91cf71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:04:30.252447 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.252356 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c20b518f-fe19-47b6-88a6-aec10c91cf71-tls-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:30.252447 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.252393 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c20b518f-fe19-47b6-88a6-aec10c91cf71-dshm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:30.252447 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.252404 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8crgx\" (UniqueName: \"kubernetes.io/projected/c20b518f-fe19-47b6-88a6-aec10c91cf71-kube-api-access-8crgx\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:30.460140 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.460097 4910 generic.go:356] "Generic (PLEG): container finished" podID="c20b518f-fe19-47b6-88a6-aec10c91cf71" containerID="e21998d7c0ef6baefefd073f3bcf21172c6058f0b104c9e22f806771ff24c0f7" exitCode=0 Oct 31 20:04:30.460369 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.460167 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" Oct 31 20:04:30.460369 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.460182 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" event={"ID":"c20b518f-fe19-47b6-88a6-aec10c91cf71","Type":"ContainerDied","Data":"e21998d7c0ef6baefefd073f3bcf21172c6058f0b104c9e22f806771ff24c0f7"} Oct 31 20:04:30.460369 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.460241 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd" event={"ID":"c20b518f-fe19-47b6-88a6-aec10c91cf71","Type":"ContainerDied","Data":"2eae7c3a46310538983d181700cf4b706044fab3573b2e9aa2d1f298a3256b82"} Oct 31 20:04:30.460369 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.460263 4910 scope.go:122] "RemoveContainer" containerID="e21998d7c0ef6baefefd073f3bcf21172c6058f0b104c9e22f806771ff24c0f7" Oct 31 20:04:30.477490 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.477473 4910 scope.go:122] "RemoveContainer" containerID="e21998d7c0ef6baefefd073f3bcf21172c6058f0b104c9e22f806771ff24c0f7" Oct 31 20:04:30.477782 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:04:30.477758 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e21998d7c0ef6baefefd073f3bcf21172c6058f0b104c9e22f806771ff24c0f7\": container with ID starting with e21998d7c0ef6baefefd073f3bcf21172c6058f0b104c9e22f806771ff24c0f7 not found: ID does not exist" containerID="e21998d7c0ef6baefefd073f3bcf21172c6058f0b104c9e22f806771ff24c0f7" Oct 31 20:04:30.477840 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.477795 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e21998d7c0ef6baefefd073f3bcf21172c6058f0b104c9e22f806771ff24c0f7"} err="failed to get container status \"e21998d7c0ef6baefefd073f3bcf21172c6058f0b104c9e22f806771ff24c0f7\": rpc error: code = NotFound desc = could not find container \"e21998d7c0ef6baefefd073f3bcf21172c6058f0b104c9e22f806771ff24c0f7\": container with ID starting with e21998d7c0ef6baefefd073f3bcf21172c6058f0b104c9e22f806771ff24c0f7 not found: ID does not exist" Oct 31 20:04:30.490370 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.490342 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd"] Oct 31 20:04:30.495271 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:30.495249 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-435728/sub-test-model-435728-kserve-859458cd67-hfwgd"] Oct 31 20:04:31.117785 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:31.117752 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body= Oct 31 20:04:31.118019 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:31.117811 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" podUID="59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" Oct 31 20:04:31.712244 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:31.712205 4910 scope.go:122] "RemoveContainer" containerID="7c27e8f35b7163da4238209bdc604f4e0d3158db105a28ba69af2faedc284f4b" Oct 31 20:04:31.741550 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:31.741525 4910 scope.go:122] "RemoveContainer" containerID="383173a8d9b8a3a84a0a66150e41fd92048e0beb4b8f71d675632ae787898861" Oct 31 20:04:31.753578 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:31.753554 4910 scope.go:122] "RemoveContainer" containerID="0b1a5f062be438ba74ec43777412648275a5cebc8af60b92aa1bc946bd97f5c0" Oct 31 20:04:31.759489 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:31.759460 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c20b518f-fe19-47b6-88a6-aec10c91cf71" path="/var/lib/kubelet/pods/c20b518f-fe19-47b6-88a6-aec10c91cf71/volumes" Oct 31 20:04:31.767869 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:31.767846 4910 scope.go:122] "RemoveContainer" containerID="66e9137467a4dee18ef419653f286177eac76183280dde096e433ffac03cd602" Oct 31 20:04:32.029291 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:32.029203 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-365b4309-n4dmb"] Oct 31 20:04:32.032944 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:32.032903 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-365b4309-n4dmb"] Oct 31 20:04:32.243372 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:32.243338 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:32.245036 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:32.245008 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:32.246833 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:32.246802 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:04:33.845676 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:33.845646 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bf64223-cc5f-4a12-99b2-af2b1248bdbe" path="/var/lib/kubelet/pods/4bf64223-cc5f-4a12-99b2-af2b1248bdbe/volumes" Oct 31 20:04:35.205336 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:35.205294 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8"] Oct 31 20:04:35.206346 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:35.205537 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" podUID="5eb49720-62f4-4d08-bad3-51b33ab7fa7f" containerName="istio-proxy" containerID="cri-o://2b67d9321bb77d833c5c4707e6a48cbe598eb73c445fa093c94d40d99f044b41" gracePeriod=30 Oct 31 20:04:40.517443 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.517417 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 20:04:40.548487 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.548456 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-envoy\") pod \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " Oct 31 20:04:40.548685 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.548547 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-workload-socket\") pod \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " Oct 31 20:04:40.548685 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.548625 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-podinfo\") pod \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " Oct 31 20:04:40.548843 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.548733 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istiod-ca-cert\") pod \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " Oct 31 20:04:40.548843 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.548806 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-kube-api-access-89wnr\" (UniqueName: \"kubernetes.io/projected/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-kube-api-access-89wnr\") pod \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " Oct 31 20:04:40.548984 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.548862 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-workload-socket" pod "5eb49720-62f4-4d08-bad3-51b33ab7fa7f" (UID: "5eb49720-62f4-4d08-bad3-51b33ab7fa7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:40.548984 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.548880 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-credential-socket\") pod \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " Oct 31 20:04:40.548984 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.548962 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-data\") pod \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " Oct 31 20:04:40.549176 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.549155 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-credential-socket" pod "5eb49720-62f4-4d08-bad3-51b33ab7fa7f" (UID: "5eb49720-62f4-4d08-bad3-51b33ab7fa7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:40.549176 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.549164 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istiod-ca-cert" pod "5eb49720-62f4-4d08-bad3-51b33ab7fa7f" (UID: "5eb49720-62f4-4d08-bad3-51b33ab7fa7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:04:40.549393 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.549372 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-token\" (UniqueName: \"kubernetes.io/projected/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-token\") pod \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " Oct 31 20:04:40.549508 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.549492 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-workload-certs\") pod \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\" (UID: \"5eb49720-62f4-4d08-bad3-51b33ab7fa7f\") " Oct 31 20:04:40.549782 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.549752 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-data" pod "5eb49720-62f4-4d08-bad3-51b33ab7fa7f" (UID: "5eb49720-62f4-4d08-bad3-51b33ab7fa7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:40.550059 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.549978 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-workload-certs" pod "5eb49720-62f4-4d08-bad3-51b33ab7fa7f" (UID: "5eb49720-62f4-4d08-bad3-51b33ab7fa7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:40.550466 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.550401 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:40.550466 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.550432 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:40.550466 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.550446 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:40.550466 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.550460 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:40.553546 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.550474 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:40.553546 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.551583 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-envoy" pod "5eb49720-62f4-4d08-bad3-51b33ab7fa7f" (UID: "5eb49720-62f4-4d08-bad3-51b33ab7fa7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:40.553546 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.553068 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-kube-api-access-89wnr" pod "5eb49720-62f4-4d08-bad3-51b33ab7fa7f" (UID: "5eb49720-62f4-4d08-bad3-51b33ab7fa7f"). InnerVolumeSpecName "kube-api-access-89wnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:40.553546 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.553487 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-podinfo" pod "5eb49720-62f4-4d08-bad3-51b33ab7fa7f" (UID: "5eb49720-62f4-4d08-bad3-51b33ab7fa7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:04:40.566362 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.566334 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-token" pod "5eb49720-62f4-4d08-bad3-51b33ab7fa7f" (UID: "5eb49720-62f4-4d08-bad3-51b33ab7fa7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:40.651687 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.651658 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:40.651687 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.651685 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:40.651959 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.651699 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89wnr\" (UniqueName: \"kubernetes.io/projected/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-kube-api-access-89wnr\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:40.651959 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.651714 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eb49720-62f4-4d08-bad3-51b33ab7fa7f-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:40.851478 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.848684 4910 generic.go:356] "Generic (PLEG): container finished" podID="5eb49720-62f4-4d08-bad3-51b33ab7fa7f" containerID="2b67d9321bb77d833c5c4707e6a48cbe598eb73c445fa093c94d40d99f044b41" exitCode=0 Oct 31 20:04:40.851478 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.848889 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" Oct 31 20:04:40.851478 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.849018 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" event={"ID":"5eb49720-62f4-4d08-bad3-51b33ab7fa7f","Type":"ContainerDied","Data":"2b67d9321bb77d833c5c4707e6a48cbe598eb73c445fa093c94d40d99f044b41"} Oct 31 20:04:40.851478 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.849071 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8" event={"ID":"5eb49720-62f4-4d08-bad3-51b33ab7fa7f","Type":"ContainerDied","Data":"67ad32e32456fd9ef242e9e8bb3f4df3ec07f9dcf12db40bc21afa7b883442b4"} Oct 31 20:04:40.851478 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.849103 4910 scope.go:122] "RemoveContainer" containerID="2b67d9321bb77d833c5c4707e6a48cbe598eb73c445fa093c94d40d99f044b41" Oct 31 20:04:40.878022 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.877984 4910 scope.go:122] "RemoveContainer" containerID="2b67d9321bb77d833c5c4707e6a48cbe598eb73c445fa093c94d40d99f044b41" Oct 31 20:04:40.878429 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:04:40.878404 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b67d9321bb77d833c5c4707e6a48cbe598eb73c445fa093c94d40d99f044b41\": container with ID starting with 2b67d9321bb77d833c5c4707e6a48cbe598eb73c445fa093c94d40d99f044b41 not found: ID does not exist" containerID="2b67d9321bb77d833c5c4707e6a48cbe598eb73c445fa093c94d40d99f044b41" Oct 31 20:04:40.878543 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.878437 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b67d9321bb77d833c5c4707e6a48cbe598eb73c445fa093c94d40d99f044b41"} err="failed to get container status \"2b67d9321bb77d833c5c4707e6a48cbe598eb73c445fa093c94d40d99f044b41\": rpc error: code = NotFound desc = could not find container \"2b67d9321bb77d833c5c4707e6a48cbe598eb73c445fa093c94d40d99f044b41\": container with ID starting with 2b67d9321bb77d833c5c4707e6a48cbe598eb73c445fa093c94d40d99f044b41 not found: ID does not exist" Oct 31 20:04:40.888410 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.888382 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8"] Oct 31 20:04:40.904014 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:40.903988 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-w8jm8"] Oct 31 20:04:40.929647 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:04:40.929615 4910 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb49720_62f4_4d08_bad3_51b33ab7fa7f.slice/crio-67ad32e32456fd9ef242e9e8bb3f4df3ec07f9dcf12db40bc21afa7b883442b4\": RecentStats: unable to find data in memory cache]" Oct 31 20:04:41.757369 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:41.757338 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eb49720-62f4-4d08-bad3-51b33ab7fa7f" path="/var/lib/kubelet/pods/5eb49720-62f4-4d08-bad3-51b33ab7fa7f/volumes" Oct 31 20:04:42.715733 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:04:42.715694 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:04:43.575771 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:43.575734 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh"] Oct 31 20:04:43.576278 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:43.576026 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" podUID="a27f1f16-7533-4079-9702-f1ed916a760f" containerName="istio-proxy" containerID="cri-o://3da1a71f7335760b52d31877b6a085968cdd8ff10a4c56790ab6cc92d60e73f8" gracePeriod=30 Oct 31 20:04:43.604424 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:43.604374 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t"] Oct 31 20:04:43.604660 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:43.604635 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" podUID="10538611-7fbc-4a9c-b7f9-61c479d57fc8" containerName="istio-proxy" containerID="cri-o://d37cbe366f1bf8e80c2ca3f859432b1d6ed6cd443573dd1d4f6c12de46ae6724" gracePeriod=30 Oct 31 20:04:44.006147 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.006067 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:04:44.006147 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.006129 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" podUID="10538611-7fbc-4a9c-b7f9-61c479d57fc8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:04:44.066491 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.066457 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5"] Oct 31 20:04:44.067106 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.067092 4910 cpu_manager.go:414] "containerMap: removing container" podUID="4936de05-9195-4a0a-b44c-75049422759b" containerName="revoke-keys" Oct 31 20:04:44.067176 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.067108 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4936de05-9195-4a0a-b44c-75049422759b" containerName="revoke-keys" Oct 31 20:04:44.067176 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.067145 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c20b518f-fe19-47b6-88a6-aec10c91cf71" containerName="main" Oct 31 20:04:44.067176 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.067150 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c20b518f-fe19-47b6-88a6-aec10c91cf71" containerName="main" Oct 31 20:04:44.067176 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.067159 4910 cpu_manager.go:414] "containerMap: removing container" podUID="5eb49720-62f4-4d08-bad3-51b33ab7fa7f" containerName="istio-proxy" Oct 31 20:04:44.067176 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.067165 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb49720-62f4-4d08-bad3-51b33ab7fa7f" containerName="istio-proxy" Oct 31 20:04:44.067342 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.067254 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="4936de05-9195-4a0a-b44c-75049422759b" containerName="revoke-keys" Oct 31 20:04:44.067342 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.067269 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c20b518f-fe19-47b6-88a6-aec10c91cf71" containerName="main" Oct 31 20:04:44.067342 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.067276 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eb49720-62f4-4d08-bad3-51b33ab7fa7f" containerName="istio-proxy" Oct 31 20:04:44.072271 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.072253 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5" Oct 31 20:04:44.078549 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.078374 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:04:44.079332 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.079309 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 20:04:44.082770 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.082750 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5"] Oct 31 20:04:44.122999 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.122957 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3d4ffab-e555-498c-b26b-dbf614031f21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5\" (UID: \"d3d4ffab-e555-498c-b26b-dbf614031f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5" Oct 31 20:04:44.123196 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.123078 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3d4ffab-e555-498c-b26b-dbf614031f21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5\" (UID: \"d3d4ffab-e555-498c-b26b-dbf614031f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5" Oct 31 20:04:44.123274 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.123253 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qr9j\" (UniqueName: \"kubernetes.io/projected/d3d4ffab-e555-498c-b26b-dbf614031f21-kube-api-access-7qr9j\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5\" (UID: \"d3d4ffab-e555-498c-b26b-dbf614031f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5" Oct 31 20:04:44.224396 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.224362 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3d4ffab-e555-498c-b26b-dbf614031f21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5\" (UID: \"d3d4ffab-e555-498c-b26b-dbf614031f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5" Oct 31 20:04:44.224605 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.224451 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3d4ffab-e555-498c-b26b-dbf614031f21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5\" (UID: \"d3d4ffab-e555-498c-b26b-dbf614031f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5" Oct 31 20:04:44.224605 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.224585 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qr9j\" (UniqueName: \"kubernetes.io/projected/d3d4ffab-e555-498c-b26b-dbf614031f21-kube-api-access-7qr9j\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5\" (UID: \"d3d4ffab-e555-498c-b26b-dbf614031f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5" Oct 31 20:04:44.225252 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.225225 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3d4ffab-e555-498c-b26b-dbf614031f21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5\" (UID: \"d3d4ffab-e555-498c-b26b-dbf614031f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5" Oct 31 20:04:44.227013 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.226984 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3d4ffab-e555-498c-b26b-dbf614031f21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5\" (UID: \"d3d4ffab-e555-498c-b26b-dbf614031f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5" Oct 31 20:04:44.238567 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.238536 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qr9j\" (UniqueName: \"kubernetes.io/projected/d3d4ffab-e555-498c-b26b-dbf614031f21-kube-api-access-7qr9j\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5\" (UID: \"d3d4ffab-e555-498c-b26b-dbf614031f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5" Oct 31 20:04:44.390160 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.390126 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5" Oct 31 20:04:44.685276 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:04:44.685215 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d4ffab_e555_498c_b26b_dbf614031f21.slice/crio-d88822eb7369364575a84b19e971d7b7de77cb53b11dbf5eaaa5247be2461f41 WatchSource:0}: Error finding container d88822eb7369364575a84b19e971d7b7de77cb53b11dbf5eaaa5247be2461f41: Status 404 returned error can't find the container with id d88822eb7369364575a84b19e971d7b7de77cb53b11dbf5eaaa5247be2461f41 Oct 31 20:04:44.690850 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.687563 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5"] Oct 31 20:04:44.910579 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:44.910540 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5" event={"ID":"d3d4ffab-e555-498c-b26b-dbf614031f21","Type":"ContainerStarted","Data":"d88822eb7369364575a84b19e971d7b7de77cb53b11dbf5eaaa5247be2461f41"} Oct 31 20:04:45.920412 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:45.920376 4910 generic.go:356] "Generic (PLEG): container finished" podID="d3d4ffab-e555-498c-b26b-dbf614031f21" containerID="a2260afb15cb0b3ef0e2a92d45210220cb807724d0dc4c57d9f55704d85fab50" exitCode=0 Oct 31 20:04:45.920842 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:45.920460 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5" event={"ID":"d3d4ffab-e555-498c-b26b-dbf614031f21","Type":"ContainerDied","Data":"a2260afb15cb0b3ef0e2a92d45210220cb807724d0dc4c57d9f55704d85fab50"} Oct 31 20:04:47.122133 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:47.122102 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5" Oct 31 20:04:47.195733 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:47.195655 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d4ffab-e555-498c-b26b-dbf614031f21-kube-api-access-7qr9j\" (UniqueName: \"kubernetes.io/projected/d3d4ffab-e555-498c-b26b-dbf614031f21-kube-api-access-7qr9j\") pod \"d3d4ffab-e555-498c-b26b-dbf614031f21\" (UID: \"d3d4ffab-e555-498c-b26b-dbf614031f21\") " Oct 31 20:04:47.195960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:47.195807 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3d4ffab-e555-498c-b26b-dbf614031f21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3d4ffab-e555-498c-b26b-dbf614031f21-maas-api-xks-ca\") pod \"d3d4ffab-e555-498c-b26b-dbf614031f21\" (UID: \"d3d4ffab-e555-498c-b26b-dbf614031f21\") " Oct 31 20:04:47.195960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:47.195834 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3d4ffab-e555-498c-b26b-dbf614031f21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3d4ffab-e555-498c-b26b-dbf614031f21-maas-api-openshift-ca\") pod \"d3d4ffab-e555-498c-b26b-dbf614031f21\" (UID: \"d3d4ffab-e555-498c-b26b-dbf614031f21\") " Oct 31 20:04:47.196260 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:47.196230 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d4ffab-e555-498c-b26b-dbf614031f21-maas-api-openshift-ca" pod "d3d4ffab-e555-498c-b26b-dbf614031f21" (UID: "d3d4ffab-e555-498c-b26b-dbf614031f21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:04:47.196453 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:47.196434 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3d4ffab-e555-498c-b26b-dbf614031f21-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:47.197941 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:47.197900 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3d4ffab-e555-498c-b26b-dbf614031f21-maas-api-xks-ca" pod "d3d4ffab-e555-498c-b26b-dbf614031f21" (UID: "d3d4ffab-e555-498c-b26b-dbf614031f21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:04:47.198027 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:47.197995 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d4ffab-e555-498c-b26b-dbf614031f21-kube-api-access-7qr9j" pod "d3d4ffab-e555-498c-b26b-dbf614031f21" (UID: "d3d4ffab-e555-498c-b26b-dbf614031f21"). InnerVolumeSpecName "kube-api-access-7qr9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:47.298056 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:47.298020 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qr9j\" (UniqueName: \"kubernetes.io/projected/d3d4ffab-e555-498c-b26b-dbf614031f21-kube-api-access-7qr9j\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:47.298056 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:47.298048 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3d4ffab-e555-498c-b26b-dbf614031f21-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:48.027935 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:48.015971 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5" event={"ID":"d3d4ffab-e555-498c-b26b-dbf614031f21","Type":"ContainerDied","Data":"d88822eb7369364575a84b19e971d7b7de77cb53b11dbf5eaaa5247be2461f41"} Oct 31 20:04:48.027935 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:48.016037 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d88822eb7369364575a84b19e971d7b7de77cb53b11dbf5eaaa5247be2461f41" Oct 31 20:04:48.027935 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:48.016118 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5" Oct 31 20:04:48.930861 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:48.930828 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 20:04:48.987595 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:48.987572 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 20:04:49.034141 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.031376 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-workload-socket\") pod \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " Oct 31 20:04:49.034141 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.031429 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-credential-socket\") pod \"a27f1f16-7533-4079-9702-f1ed916a760f\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " Oct 31 20:04:49.034141 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.031460 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istiod-ca-cert\") pod \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " Oct 31 20:04:49.034141 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.031499 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-istio-data\") pod \"a27f1f16-7533-4079-9702-f1ed916a760f\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " Oct 31 20:04:49.034141 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.031611 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-credential-socket\") pod \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " Oct 31 20:04:49.034141 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.031656 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-workload-socket\") pod \"a27f1f16-7533-4079-9702-f1ed916a760f\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " Oct 31 20:04:49.034141 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.031691 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a27f1f16-7533-4079-9702-f1ed916a760f-istio-token\" (UniqueName: \"kubernetes.io/projected/a27f1f16-7533-4079-9702-f1ed916a760f-istio-token\") pod \"a27f1f16-7533-4079-9702-f1ed916a760f\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " Oct 31 20:04:49.034141 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.031727 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-workload-certs\") pod \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " Oct 31 20:04:49.034141 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.031756 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-token\" (UniqueName: \"kubernetes.io/projected/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-token\") pod \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " Oct 31 20:04:49.034141 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.031796 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a27f1f16-7533-4079-9702-f1ed916a760f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a27f1f16-7533-4079-9702-f1ed916a760f-istiod-ca-cert\") pod \"a27f1f16-7533-4079-9702-f1ed916a760f\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " Oct 31 20:04:49.034141 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.031865 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-data\") pod \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " Oct 31 20:04:49.034141 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.031934 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a27f1f16-7533-4079-9702-f1ed916a760f-kube-api-access-mpf2l\" (UniqueName: \"kubernetes.io/projected/a27f1f16-7533-4079-9702-f1ed916a760f-kube-api-access-mpf2l\") pod \"a27f1f16-7533-4079-9702-f1ed916a760f\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " Oct 31 20:04:49.034141 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.031973 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-podinfo\") pod \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " Oct 31 20:04:49.034141 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.032005 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-workload-certs\") pod \"a27f1f16-7533-4079-9702-f1ed916a760f\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " Oct 31 20:04:49.034141 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.032068 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-istio-envoy\") pod \"a27f1f16-7533-4079-9702-f1ed916a760f\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " Oct 31 20:04:49.034141 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.032147 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10538611-7fbc-4a9c-b7f9-61c479d57fc8-kube-api-access-bnlkn\" (UniqueName: \"kubernetes.io/projected/10538611-7fbc-4a9c-b7f9-61c479d57fc8-kube-api-access-bnlkn\") pod \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " Oct 31 20:04:49.034141 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.032261 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a27f1f16-7533-4079-9702-f1ed916a760f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a27f1f16-7533-4079-9702-f1ed916a760f-istio-podinfo\") pod \"a27f1f16-7533-4079-9702-f1ed916a760f\" (UID: \"a27f1f16-7533-4079-9702-f1ed916a760f\") " Oct 31 20:04:49.034141 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.032331 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-envoy\") pod \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\" (UID: \"10538611-7fbc-4a9c-b7f9-61c479d57fc8\") " Oct 31 20:04:49.040116 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.038903 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-workload-certs" pod "10538611-7fbc-4a9c-b7f9-61c479d57fc8" (UID: "10538611-7fbc-4a9c-b7f9-61c479d57fc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:49.089931 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.085323 4910 generic.go:356] "Generic (PLEG): container finished" podID="10538611-7fbc-4a9c-b7f9-61c479d57fc8" containerID="d37cbe366f1bf8e80c2ca3f859432b1d6ed6cd443573dd1d4f6c12de46ae6724" exitCode=0 Oct 31 20:04:49.093603 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.087638 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-workload-certs" pod "a27f1f16-7533-4079-9702-f1ed916a760f" (UID: "a27f1f16-7533-4079-9702-f1ed916a760f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:49.093804 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.088059 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a27f1f16-7533-4079-9702-f1ed916a760f-istiod-ca-cert" pod "a27f1f16-7533-4079-9702-f1ed916a760f" (UID: "a27f1f16-7533-4079-9702-f1ed916a760f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:04:49.093888 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.088299 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-data" pod "10538611-7fbc-4a9c-b7f9-61c479d57fc8" (UID: "10538611-7fbc-4a9c-b7f9-61c479d57fc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:49.094010 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.090190 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" event={"ID":"10538611-7fbc-4a9c-b7f9-61c479d57fc8","Type":"ContainerDied","Data":"d37cbe366f1bf8e80c2ca3f859432b1d6ed6cd443573dd1d4f6c12de46ae6724"} Oct 31 20:04:49.094221 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.094192 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" event={"ID":"10538611-7fbc-4a9c-b7f9-61c479d57fc8","Type":"ContainerDied","Data":"7e234628d0f7f7bbb31fc14a8d2934d5f3984a9646b8334e5ac4a1ab65fe3b27"} Oct 31 20:04:49.094367 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.094340 4910 scope.go:122] "RemoveContainer" containerID="d37cbe366f1bf8e80c2ca3f859432b1d6ed6cd443573dd1d4f6c12de46ae6724" Oct 31 20:04:49.094613 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.090298 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t" Oct 31 20:04:49.097292 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.096149 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-istio-data" pod "a27f1f16-7533-4079-9702-f1ed916a760f" (UID: "a27f1f16-7533-4079-9702-f1ed916a760f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:49.097292 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.096511 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-workload-socket" pod "10538611-7fbc-4a9c-b7f9-61c479d57fc8" (UID: "10538611-7fbc-4a9c-b7f9-61c479d57fc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:49.097292 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.096703 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-credential-socket" pod "a27f1f16-7533-4079-9702-f1ed916a760f" (UID: "a27f1f16-7533-4079-9702-f1ed916a760f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:49.097292 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.096995 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istiod-ca-cert" pod "10538611-7fbc-4a9c-b7f9-61c479d57fc8" (UID: "10538611-7fbc-4a9c-b7f9-61c479d57fc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:04:49.101423 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.098742 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-credential-socket" pod "10538611-7fbc-4a9c-b7f9-61c479d57fc8" (UID: "10538611-7fbc-4a9c-b7f9-61c479d57fc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:49.101423 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.098846 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-workload-socket" pod "a27f1f16-7533-4079-9702-f1ed916a760f" (UID: "a27f1f16-7533-4079-9702-f1ed916a760f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:49.101423 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.100861 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-envoy" pod "10538611-7fbc-4a9c-b7f9-61c479d57fc8" (UID: "10538611-7fbc-4a9c-b7f9-61c479d57fc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:49.102345 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.102313 4910 generic.go:356] "Generic (PLEG): container finished" podID="a27f1f16-7533-4079-9702-f1ed916a760f" containerID="3da1a71f7335760b52d31877b6a085968cdd8ff10a4c56790ab6cc92d60e73f8" exitCode=0 Oct 31 20:04:49.102480 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.102461 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" event={"ID":"a27f1f16-7533-4079-9702-f1ed916a760f","Type":"ContainerDied","Data":"3da1a71f7335760b52d31877b6a085968cdd8ff10a4c56790ab6cc92d60e73f8"} Oct 31 20:04:49.102555 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.102512 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" event={"ID":"a27f1f16-7533-4079-9702-f1ed916a760f","Type":"ContainerDied","Data":"5454af44879cb4ff131d50103dbe876f9e2acbca2ed1c5402921c0372e98f9b3"} Oct 31 20:04:49.102623 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.102601 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" Oct 31 20:04:49.102738 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.102707 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a27f1f16-7533-4079-9702-f1ed916a760f-istio-token" pod "a27f1f16-7533-4079-9702-f1ed916a760f" (UID: "a27f1f16-7533-4079-9702-f1ed916a760f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:49.105509 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.105456 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-podinfo" pod "10538611-7fbc-4a9c-b7f9-61c479d57fc8" (UID: "10538611-7fbc-4a9c-b7f9-61c479d57fc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:04:49.105791 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.105746 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-istio-envoy" pod "a27f1f16-7533-4079-9702-f1ed916a760f" (UID: "a27f1f16-7533-4079-9702-f1ed916a760f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:04:49.106878 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.106822 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a27f1f16-7533-4079-9702-f1ed916a760f-istio-podinfo" pod "a27f1f16-7533-4079-9702-f1ed916a760f" (UID: "a27f1f16-7533-4079-9702-f1ed916a760f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:04:49.108495 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.108473 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a27f1f16-7533-4079-9702-f1ed916a760f-kube-api-access-mpf2l" pod "a27f1f16-7533-4079-9702-f1ed916a760f" (UID: "a27f1f16-7533-4079-9702-f1ed916a760f"). InnerVolumeSpecName "kube-api-access-mpf2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:49.109301 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.109281 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-token" pod "10538611-7fbc-4a9c-b7f9-61c479d57fc8" (UID: "10538611-7fbc-4a9c-b7f9-61c479d57fc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:49.109742 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.109711 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10538611-7fbc-4a9c-b7f9-61c479d57fc8-kube-api-access-bnlkn" pod "10538611-7fbc-4a9c-b7f9-61c479d57fc8" (UID: "10538611-7fbc-4a9c-b7f9-61c479d57fc8"). InnerVolumeSpecName "kube-api-access-bnlkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:49.121957 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.118358 4910 scope.go:122] "RemoveContainer" containerID="d37cbe366f1bf8e80c2ca3f859432b1d6ed6cd443573dd1d4f6c12de46ae6724" Oct 31 20:04:49.125387 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:04:49.125359 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37cbe366f1bf8e80c2ca3f859432b1d6ed6cd443573dd1d4f6c12de46ae6724\": container with ID starting with d37cbe366f1bf8e80c2ca3f859432b1d6ed6cd443573dd1d4f6c12de46ae6724 not found: ID does not exist" containerID="d37cbe366f1bf8e80c2ca3f859432b1d6ed6cd443573dd1d4f6c12de46ae6724" Oct 31 20:04:49.125497 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.125402 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37cbe366f1bf8e80c2ca3f859432b1d6ed6cd443573dd1d4f6c12de46ae6724"} err="failed to get container status \"d37cbe366f1bf8e80c2ca3f859432b1d6ed6cd443573dd1d4f6c12de46ae6724\": rpc error: code = NotFound desc = could not find container \"d37cbe366f1bf8e80c2ca3f859432b1d6ed6cd443573dd1d4f6c12de46ae6724\": container with ID starting with d37cbe366f1bf8e80c2ca3f859432b1d6ed6cd443573dd1d4f6c12de46ae6724 not found: ID does not exist" Oct 31 20:04:49.125497 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.125428 4910 scope.go:122] "RemoveContainer" containerID="3da1a71f7335760b52d31877b6a085968cdd8ff10a4c56790ab6cc92d60e73f8" Oct 31 20:04:49.137695 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.137666 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:49.137695 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.137693 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:49.137876 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.137709 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:49.137876 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.137718 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:49.137876 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.137726 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:49.137876 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.137735 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:49.137876 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.137743 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a27f1f16-7533-4079-9702-f1ed916a760f-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:49.137876 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.137751 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:49.137876 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.137759 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:49.137876 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.137767 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a27f1f16-7533-4079-9702-f1ed916a760f-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:49.137876 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.137777 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:49.137876 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.137789 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpf2l\" (UniqueName: \"kubernetes.io/projected/a27f1f16-7533-4079-9702-f1ed916a760f-kube-api-access-mpf2l\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:49.137876 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.137799 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:49.137876 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.137807 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:49.137876 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.137815 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a27f1f16-7533-4079-9702-f1ed916a760f-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:49.137876 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.137823 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnlkn\" (UniqueName: \"kubernetes.io/projected/10538611-7fbc-4a9c-b7f9-61c479d57fc8-kube-api-access-bnlkn\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:49.137876 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.137833 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a27f1f16-7533-4079-9702-f1ed916a760f-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:49.137876 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.137841 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10538611-7fbc-4a9c-b7f9-61c479d57fc8-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:49.141896 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.141860 4910 scope.go:122] "RemoveContainer" containerID="3da1a71f7335760b52d31877b6a085968cdd8ff10a4c56790ab6cc92d60e73f8" Oct 31 20:04:49.142299 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:04:49.142247 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3da1a71f7335760b52d31877b6a085968cdd8ff10a4c56790ab6cc92d60e73f8\": container with ID starting with 3da1a71f7335760b52d31877b6a085968cdd8ff10a4c56790ab6cc92d60e73f8 not found: ID does not exist" containerID="3da1a71f7335760b52d31877b6a085968cdd8ff10a4c56790ab6cc92d60e73f8" Oct 31 20:04:49.142387 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.142289 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3da1a71f7335760b52d31877b6a085968cdd8ff10a4c56790ab6cc92d60e73f8"} err="failed to get container status \"3da1a71f7335760b52d31877b6a085968cdd8ff10a4c56790ab6cc92d60e73f8\": rpc error: code = NotFound desc = could not find container \"3da1a71f7335760b52d31877b6a085968cdd8ff10a4c56790ab6cc92d60e73f8\": container with ID starting with 3da1a71f7335760b52d31877b6a085968cdd8ff10a4c56790ab6cc92d60e73f8 not found: ID does not exist" Oct 31 20:04:49.434013 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.433932 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t"] Oct 31 20:04:49.438850 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.438804 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-86g8t"] Oct 31 20:04:49.466865 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.466829 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh"] Oct 31 20:04:49.476176 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.476143 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh"] Oct 31 20:04:49.757325 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.757245 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10538611-7fbc-4a9c-b7f9-61c479d57fc8" path="/var/lib/kubelet/pods/10538611-7fbc-4a9c-b7f9-61c479d57fc8/volumes" Oct 31 20:04:49.758074 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.758051 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a27f1f16-7533-4079-9702-f1ed916a760f" path="/var/lib/kubelet/pods/a27f1f16-7533-4079-9702-f1ed916a760f/volumes" Oct 31 20:04:49.788753 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.788719 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:04:49.788941 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:49.788773 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-24xrh" podUID="a27f1f16-7533-4079-9702-f1ed916a760f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:04:55.388258 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:55.376610 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp"] Oct 31 20:04:55.388258 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:55.376857 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp" podUID="16950395-c37f-4c55-bc84-6be3d07c91f6" containerName="maas-api" containerID="cri-o://cb5202b2ec1528bc21f48fdd1a67957a6403d3d6dd02f468eeff069f37432934" gracePeriod=30 Oct 31 20:04:55.903042 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:55.903003 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5"] Oct 31 20:04:55.947749 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:55.947674 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-d9c048-6jbv5"] Oct 31 20:04:56.047344 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:56.047316 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp" Oct 31 20:04:56.161747 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:56.161643 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16950395-c37f-4c55-bc84-6be3d07c91f6-kube-api-access-4mlqq\" (UniqueName: \"kubernetes.io/projected/16950395-c37f-4c55-bc84-6be3d07c91f6-kube-api-access-4mlqq\") pod \"16950395-c37f-4c55-bc84-6be3d07c91f6\" (UID: \"16950395-c37f-4c55-bc84-6be3d07c91f6\") " Oct 31 20:04:56.162000 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:56.161833 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16950395-c37f-4c55-bc84-6be3d07c91f6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/16950395-c37f-4c55-bc84-6be3d07c91f6-metrics-certs\") pod \"16950395-c37f-4c55-bc84-6be3d07c91f6\" (UID: \"16950395-c37f-4c55-bc84-6be3d07c91f6\") " Oct 31 20:04:56.162000 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:56.161859 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16950395-c37f-4c55-bc84-6be3d07c91f6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16950395-c37f-4c55-bc84-6be3d07c91f6-openshift-service-ca\") pod \"16950395-c37f-4c55-bc84-6be3d07c91f6\" (UID: \"16950395-c37f-4c55-bc84-6be3d07c91f6\") " Oct 31 20:04:56.162000 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:56.161890 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16950395-c37f-4c55-bc84-6be3d07c91f6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16950395-c37f-4c55-bc84-6be3d07c91f6-maas-api-tls\") pod \"16950395-c37f-4c55-bc84-6be3d07c91f6\" (UID: \"16950395-c37f-4c55-bc84-6be3d07c91f6\") " Oct 31 20:04:56.162352 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:56.162323 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16950395-c37f-4c55-bc84-6be3d07c91f6-openshift-service-ca" pod "16950395-c37f-4c55-bc84-6be3d07c91f6" (UID: "16950395-c37f-4c55-bc84-6be3d07c91f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:04:56.162552 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:56.162531 4910 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16950395-c37f-4c55-bc84-6be3d07c91f6-openshift-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:56.164333 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:56.164294 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16950395-c37f-4c55-bc84-6be3d07c91f6-maas-api-tls" pod "16950395-c37f-4c55-bc84-6be3d07c91f6" (UID: "16950395-c37f-4c55-bc84-6be3d07c91f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:04:56.164333 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:56.164320 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16950395-c37f-4c55-bc84-6be3d07c91f6-metrics-certs" pod "16950395-c37f-4c55-bc84-6be3d07c91f6" (UID: "16950395-c37f-4c55-bc84-6be3d07c91f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:04:56.164537 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:56.164456 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16950395-c37f-4c55-bc84-6be3d07c91f6-kube-api-access-4mlqq" pod "16950395-c37f-4c55-bc84-6be3d07c91f6" (UID: "16950395-c37f-4c55-bc84-6be3d07c91f6"). InnerVolumeSpecName "kube-api-access-4mlqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:04:56.264321 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:56.264283 4910 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16950395-c37f-4c55-bc84-6be3d07c91f6-metrics-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:56.264321 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:56.264316 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16950395-c37f-4c55-bc84-6be3d07c91f6-maas-api-tls\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:56.264321 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:56.264327 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mlqq\" (UniqueName: \"kubernetes.io/projected/16950395-c37f-4c55-bc84-6be3d07c91f6-kube-api-access-4mlqq\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:04:56.290428 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:56.290393 4910 generic.go:356] "Generic (PLEG): container finished" podID="16950395-c37f-4c55-bc84-6be3d07c91f6" containerID="cb5202b2ec1528bc21f48fdd1a67957a6403d3d6dd02f468eeff069f37432934" exitCode=0 Oct 31 20:04:56.290636 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:56.290453 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp" Oct 31 20:04:56.290636 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:56.290480 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp" event={"ID":"16950395-c37f-4c55-bc84-6be3d07c91f6","Type":"ContainerDied","Data":"cb5202b2ec1528bc21f48fdd1a67957a6403d3d6dd02f468eeff069f37432934"} Oct 31 20:04:56.290636 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:56.290576 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp" event={"ID":"16950395-c37f-4c55-bc84-6be3d07c91f6","Type":"ContainerDied","Data":"8f38ec94b53c3ecfb600237a996da4602dda83bcd8fc7366d3eac2c77c7729d0"} Oct 31 20:04:56.290636 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:56.290606 4910 scope.go:122] "RemoveContainer" containerID="cb5202b2ec1528bc21f48fdd1a67957a6403d3d6dd02f468eeff069f37432934" Oct 31 20:04:56.302487 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:56.302472 4910 scope.go:122] "RemoveContainer" containerID="cb5202b2ec1528bc21f48fdd1a67957a6403d3d6dd02f468eeff069f37432934" Oct 31 20:04:56.302736 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:04:56.302714 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb5202b2ec1528bc21f48fdd1a67957a6403d3d6dd02f468eeff069f37432934\": container with ID starting with cb5202b2ec1528bc21f48fdd1a67957a6403d3d6dd02f468eeff069f37432934 not found: ID does not exist" containerID="cb5202b2ec1528bc21f48fdd1a67957a6403d3d6dd02f468eeff069f37432934" Oct 31 20:04:56.302790 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:56.302743 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb5202b2ec1528bc21f48fdd1a67957a6403d3d6dd02f468eeff069f37432934"} err="failed to get container status \"cb5202b2ec1528bc21f48fdd1a67957a6403d3d6dd02f468eeff069f37432934\": rpc error: code = NotFound desc = could not find container \"cb5202b2ec1528bc21f48fdd1a67957a6403d3d6dd02f468eeff069f37432934\": container with ID starting with cb5202b2ec1528bc21f48fdd1a67957a6403d3d6dd02f468eeff069f37432934 not found: ID does not exist" Oct 31 20:04:56.323497 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:56.323468 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp"] Oct 31 20:04:56.327302 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:56.327280 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d9c048-7b44cbccd5-m99fp"] Oct 31 20:04:57.756988 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:57.756951 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16950395-c37f-4c55-bc84-6be3d07c91f6" path="/var/lib/kubelet/pods/16950395-c37f-4c55-bc84-6be3d07c91f6/volumes" Oct 31 20:04:57.757526 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:04:57.757510 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3d4ffab-e555-498c-b26b-dbf614031f21" path="/var/lib/kubelet/pods/d3d4ffab-e555-498c-b26b-dbf614031f21/volumes" Oct 31 20:05:05.286106 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:05.285036 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7"] Oct 31 20:05:05.286106 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:05.285334 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" podUID="64efa0c3-abc5-4253-8f01-cb9954e33dc6" containerName="istio-proxy" containerID="cri-o://e722a46bfbd56c97023369be32ecbd1a3382095ec2e5c1d9c950aff77459dc7d" gracePeriod=30 Oct 31 20:05:05.291015 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:05.290986 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf"] Oct 31 20:05:05.291338 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:05.291309 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" podUID="4a521083-7066-42d2-8997-7e97d4a7ef3e" containerName="istio-proxy" containerID="cri-o://998030f55119599ed804de17e403d16183f6e93439c046fd91588ed5a7148824" gracePeriod=30 Oct 31 20:05:05.910478 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:05.910387 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5"] Oct 31 20:05:05.911046 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:05.911029 4910 cpu_manager.go:414] "containerMap: removing container" podUID="a27f1f16-7533-4079-9702-f1ed916a760f" containerName="istio-proxy" Oct 31 20:05:05.911046 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:05.911046 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a27f1f16-7533-4079-9702-f1ed916a760f" containerName="istio-proxy" Oct 31 20:05:05.911206 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:05.911069 4910 cpu_manager.go:414] "containerMap: removing container" podUID="16950395-c37f-4c55-bc84-6be3d07c91f6" containerName="maas-api" Oct 31 20:05:05.911206 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:05.911074 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16950395-c37f-4c55-bc84-6be3d07c91f6" containerName="maas-api" Oct 31 20:05:05.911206 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:05.911112 4910 cpu_manager.go:414] "containerMap: removing container" podUID="10538611-7fbc-4a9c-b7f9-61c479d57fc8" containerName="istio-proxy" Oct 31 20:05:05.911206 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:05.911118 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10538611-7fbc-4a9c-b7f9-61c479d57fc8" containerName="istio-proxy" Oct 31 20:05:05.911206 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:05.911127 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d3d4ffab-e555-498c-b26b-dbf614031f21" containerName="revoke-keys" Oct 31 20:05:05.911206 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:05.911131 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d4ffab-e555-498c-b26b-dbf614031f21" containerName="revoke-keys" Oct 31 20:05:05.911520 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:05.911241 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="16950395-c37f-4c55-bc84-6be3d07c91f6" containerName="maas-api" Oct 31 20:05:05.911520 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:05.911261 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="10538611-7fbc-4a9c-b7f9-61c479d57fc8" containerName="istio-proxy" Oct 31 20:05:05.911520 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:05.911268 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d4ffab-e555-498c-b26b-dbf614031f21" containerName="revoke-keys" Oct 31 20:05:05.911520 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:05.911276 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="a27f1f16-7533-4079-9702-f1ed916a760f" containerName="istio-proxy" Oct 31 20:05:05.915976 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:05.915944 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5" Oct 31 20:05:05.922735 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:05.922670 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 20:05:05.923412 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:05.923390 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:05:05.925066 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:05.925033 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5"] Oct 31 20:05:06.008723 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:06.008680 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn5jb\" (UniqueName: \"kubernetes.io/projected/a1234fff-2cf6-40b7-ab83-bd37f5a75369-kube-api-access-hn5jb\") pod \"maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5\" (UID: \"a1234fff-2cf6-40b7-ab83-bd37f5a75369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5" Oct 31 20:05:06.008895 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:06.008745 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1234fff-2cf6-40b7-ab83-bd37f5a75369-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5\" (UID: \"a1234fff-2cf6-40b7-ab83-bd37f5a75369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5" Oct 31 20:05:06.008895 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:06.008869 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1234fff-2cf6-40b7-ab83-bd37f5a75369-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5\" (UID: \"a1234fff-2cf6-40b7-ab83-bd37f5a75369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5" Oct 31 20:05:06.110672 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:06.110635 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn5jb\" (UniqueName: \"kubernetes.io/projected/a1234fff-2cf6-40b7-ab83-bd37f5a75369-kube-api-access-hn5jb\") pod \"maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5\" (UID: \"a1234fff-2cf6-40b7-ab83-bd37f5a75369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5" Oct 31 20:05:06.110867 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:06.110697 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1234fff-2cf6-40b7-ab83-bd37f5a75369-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5\" (UID: \"a1234fff-2cf6-40b7-ab83-bd37f5a75369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5" Oct 31 20:05:06.110867 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:06.110740 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1234fff-2cf6-40b7-ab83-bd37f5a75369-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5\" (UID: \"a1234fff-2cf6-40b7-ab83-bd37f5a75369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5" Oct 31 20:05:06.111451 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:06.111422 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1234fff-2cf6-40b7-ab83-bd37f5a75369-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5\" (UID: \"a1234fff-2cf6-40b7-ab83-bd37f5a75369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5" Oct 31 20:05:06.113301 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:06.113279 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1234fff-2cf6-40b7-ab83-bd37f5a75369-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5\" (UID: \"a1234fff-2cf6-40b7-ab83-bd37f5a75369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5" Oct 31 20:05:06.127023 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:06.126993 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn5jb\" (UniqueName: \"kubernetes.io/projected/a1234fff-2cf6-40b7-ab83-bd37f5a75369-kube-api-access-hn5jb\") pod \"maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5\" (UID: \"a1234fff-2cf6-40b7-ab83-bd37f5a75369\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5" Oct 31 20:05:06.248616 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:06.248519 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5" Oct 31 20:05:06.537407 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:05:06.537369 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1234fff_2cf6_40b7_ab83_bd37f5a75369.slice/crio-95b8003a80196f390cb78d10e036b7cee9af19a4925ad28ad92a18fa296d0e7f WatchSource:0}: Error finding container 95b8003a80196f390cb78d10e036b7cee9af19a4925ad28ad92a18fa296d0e7f: Status 404 returned error can't find the container with id 95b8003a80196f390cb78d10e036b7cee9af19a4925ad28ad92a18fa296d0e7f Oct 31 20:05:06.537407 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:06.537382 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5"] Oct 31 20:05:06.633829 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:06.633779 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5" event={"ID":"a1234fff-2cf6-40b7-ab83-bd37f5a75369","Type":"ContainerStarted","Data":"95b8003a80196f390cb78d10e036b7cee9af19a4925ad28ad92a18fa296d0e7f"} Oct 31 20:05:07.643760 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:07.643731 4910 generic.go:356] "Generic (PLEG): container finished" podID="a1234fff-2cf6-40b7-ab83-bd37f5a75369" containerID="2d9f858c83a797f3d13edd4bc8b583d9348df4516215e54b5b3e85956ac1a407" exitCode=0 Oct 31 20:05:07.644138 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:07.643822 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5" event={"ID":"a1234fff-2cf6-40b7-ab83-bd37f5a75369","Type":"ContainerDied","Data":"2d9f858c83a797f3d13edd4bc8b583d9348df4516215e54b5b3e85956ac1a407"} Oct 31 20:05:07.879769 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:07.879684 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:05:07.879769 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:07.879747 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" podUID="4a521083-7066-42d2-8997-7e97d4a7ef3e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:05:08.847158 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:08.847126 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5" Oct 31 20:05:08.878771 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:08.878503 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1234fff-2cf6-40b7-ab83-bd37f5a75369-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1234fff-2cf6-40b7-ab83-bd37f5a75369-maas-api-openshift-ca\") pod \"a1234fff-2cf6-40b7-ab83-bd37f5a75369\" (UID: \"a1234fff-2cf6-40b7-ab83-bd37f5a75369\") " Oct 31 20:05:08.878771 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:08.878679 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1234fff-2cf6-40b7-ab83-bd37f5a75369-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1234fff-2cf6-40b7-ab83-bd37f5a75369-maas-api-xks-ca\") pod \"a1234fff-2cf6-40b7-ab83-bd37f5a75369\" (UID: \"a1234fff-2cf6-40b7-ab83-bd37f5a75369\") " Oct 31 20:05:08.879726 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:08.878849 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1234fff-2cf6-40b7-ab83-bd37f5a75369-kube-api-access-hn5jb\" (UniqueName: \"kubernetes.io/projected/a1234fff-2cf6-40b7-ab83-bd37f5a75369-kube-api-access-hn5jb\") pod \"a1234fff-2cf6-40b7-ab83-bd37f5a75369\" (UID: \"a1234fff-2cf6-40b7-ab83-bd37f5a75369\") " Oct 31 20:05:08.879726 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:08.879508 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1234fff-2cf6-40b7-ab83-bd37f5a75369-maas-api-openshift-ca" pod "a1234fff-2cf6-40b7-ab83-bd37f5a75369" (UID: "a1234fff-2cf6-40b7-ab83-bd37f5a75369"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:05:08.882270 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:08.882210 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1234fff-2cf6-40b7-ab83-bd37f5a75369-kube-api-access-hn5jb" pod "a1234fff-2cf6-40b7-ab83-bd37f5a75369" (UID: "a1234fff-2cf6-40b7-ab83-bd37f5a75369"). InnerVolumeSpecName "kube-api-access-hn5jb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:08.882270 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:08.882227 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1234fff-2cf6-40b7-ab83-bd37f5a75369-maas-api-xks-ca" pod "a1234fff-2cf6-40b7-ab83-bd37f5a75369" (UID: "a1234fff-2cf6-40b7-ab83-bd37f5a75369"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:05:08.981166 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:08.981070 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1234fff-2cf6-40b7-ab83-bd37f5a75369-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:08.981166 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:08.981102 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn5jb\" (UniqueName: \"kubernetes.io/projected/a1234fff-2cf6-40b7-ab83-bd37f5a75369-kube-api-access-hn5jb\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:08.981166 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:08.981115 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1234fff-2cf6-40b7-ab83-bd37f5a75369-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:09.665073 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:09.665034 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5" Oct 31 20:05:09.665298 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:09.665072 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5" event={"ID":"a1234fff-2cf6-40b7-ab83-bd37f5a75369","Type":"ContainerDied","Data":"95b8003a80196f390cb78d10e036b7cee9af19a4925ad28ad92a18fa296d0e7f"} Oct 31 20:05:09.665298 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:09.665120 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95b8003a80196f390cb78d10e036b7cee9af19a4925ad28ad92a18fa296d0e7f" Oct 31 20:05:10.693726 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.693685 4910 generic.go:356] "Generic (PLEG): container finished" podID="4a521083-7066-42d2-8997-7e97d4a7ef3e" containerID="998030f55119599ed804de17e403d16183f6e93439c046fd91588ed5a7148824" exitCode=0 Oct 31 20:05:10.694358 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.694037 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" event={"ID":"4a521083-7066-42d2-8997-7e97d4a7ef3e","Type":"ContainerDied","Data":"998030f55119599ed804de17e403d16183f6e93439c046fd91588ed5a7148824"} Oct 31 20:05:10.698834 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.698803 4910 generic.go:356] "Generic (PLEG): container finished" podID="64efa0c3-abc5-4253-8f01-cb9954e33dc6" containerID="e722a46bfbd56c97023369be32ecbd1a3382095ec2e5c1d9c950aff77459dc7d" exitCode=0 Oct 31 20:05:10.699079 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.699060 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" event={"ID":"64efa0c3-abc5-4253-8f01-cb9954e33dc6","Type":"ContainerDied","Data":"e722a46bfbd56c97023369be32ecbd1a3382095ec2e5c1d9c950aff77459dc7d"} Oct 31 20:05:10.699179 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.699131 4910 scope.go:122] "RemoveContainer" containerID="f916b44d93c7fa872d1ae3cf94864bdcb7690d0977fabdc11b5e37bad0588312" Oct 31 20:05:10.735056 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.735025 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 20:05:10.737919 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.737880 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt"] Oct 31 20:05:10.761897 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.760251 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv"] Oct 31 20:05:10.786069 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.786028 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-token\" (UniqueName: \"kubernetes.io/projected/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-token\") pod \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " Oct 31 20:05:10.789347 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.789318 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-credential-socket\") pod \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " Oct 31 20:05:10.789611 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.789592 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64efa0c3-abc5-4253-8f01-cb9954e33dc6-kube-api-access-h2ccc\" (UniqueName: \"kubernetes.io/projected/64efa0c3-abc5-4253-8f01-cb9954e33dc6-kube-api-access-h2ccc\") pod \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " Oct 31 20:05:10.789798 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.789782 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-data\") pod \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " Oct 31 20:05:10.798207 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.797031 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-workload-socket\") pod \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " Oct 31 20:05:10.798207 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.797102 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-podinfo\") pod \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " Oct 31 20:05:10.798207 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.797211 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istiod-ca-cert\") pod \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " Oct 31 20:05:10.798207 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.797371 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-envoy\") pod \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " Oct 31 20:05:10.798207 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.789035 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-token" pod "64efa0c3-abc5-4253-8f01-cb9954e33dc6" (UID: "64efa0c3-abc5-4253-8f01-cb9954e33dc6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:10.798207 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.790352 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-credential-socket" pod "64efa0c3-abc5-4253-8f01-cb9954e33dc6" (UID: "64efa0c3-abc5-4253-8f01-cb9954e33dc6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:10.798207 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.790694 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-data" pod "64efa0c3-abc5-4253-8f01-cb9954e33dc6" (UID: "64efa0c3-abc5-4253-8f01-cb9954e33dc6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:10.798207 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.797585 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-workload-socket" pod "64efa0c3-abc5-4253-8f01-cb9954e33dc6" (UID: "64efa0c3-abc5-4253-8f01-cb9954e33dc6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:10.798207 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.797804 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-workload-certs\") pod \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\" (UID: \"64efa0c3-abc5-4253-8f01-cb9954e33dc6\") " Oct 31 20:05:10.798207 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.797959 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istiod-ca-cert" pod "64efa0c3-abc5-4253-8f01-cb9954e33dc6" (UID: "64efa0c3-abc5-4253-8f01-cb9954e33dc6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:05:10.799180 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.798491 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:10.799180 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.798516 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:10.799180 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.798531 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:10.799180 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.798559 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:10.799180 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.798574 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:10.799180 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.798779 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-workload-certs" pod "64efa0c3-abc5-4253-8f01-cb9954e33dc6" (UID: "64efa0c3-abc5-4253-8f01-cb9954e33dc6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:10.804509 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.804481 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64efa0c3-abc5-4253-8f01-cb9954e33dc6-kube-api-access-h2ccc" pod "64efa0c3-abc5-4253-8f01-cb9954e33dc6" (UID: "64efa0c3-abc5-4253-8f01-cb9954e33dc6"). InnerVolumeSpecName "kube-api-access-h2ccc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:10.804953 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.804892 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-podinfo" pod "64efa0c3-abc5-4253-8f01-cb9954e33dc6" (UID: "64efa0c3-abc5-4253-8f01-cb9954e33dc6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:05:10.805038 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.805011 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-envoy" pod "64efa0c3-abc5-4253-8f01-cb9954e33dc6" (UID: "64efa0c3-abc5-4253-8f01-cb9954e33dc6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:10.858940 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.858900 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 20:05:10.918049 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.910223 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:10.918049 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.910263 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:10.918049 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.910278 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64efa0c3-abc5-4253-8f01-cb9954e33dc6-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:10.918049 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:10.910291 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2ccc\" (UniqueName: \"kubernetes.io/projected/64efa0c3-abc5-4253-8f01-cb9954e33dc6-kube-api-access-h2ccc\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:11.011342 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.011305 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-data\") pod \"4a521083-7066-42d2-8997-7e97d4a7ef3e\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " Oct 31 20:05:11.011602 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.011360 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-workload-socket\") pod \"4a521083-7066-42d2-8997-7e97d4a7ef3e\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " Oct 31 20:05:11.011602 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.011381 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-envoy\") pod \"4a521083-7066-42d2-8997-7e97d4a7ef3e\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " Oct 31 20:05:11.011602 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.011404 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a521083-7066-42d2-8997-7e97d4a7ef3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a521083-7066-42d2-8997-7e97d4a7ef3e-istiod-ca-cert\") pod \"4a521083-7066-42d2-8997-7e97d4a7ef3e\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " Oct 31 20:05:11.011602 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.011488 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-credential-socket\") pod \"4a521083-7066-42d2-8997-7e97d4a7ef3e\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " Oct 31 20:05:11.011602 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.011521 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a521083-7066-42d2-8997-7e97d4a7ef3e-kube-api-access-4x4wr\" (UniqueName: \"kubernetes.io/projected/4a521083-7066-42d2-8997-7e97d4a7ef3e-kube-api-access-4x4wr\") pod \"4a521083-7066-42d2-8997-7e97d4a7ef3e\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " Oct 31 20:05:11.011602 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.011573 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-token\" (UniqueName: \"kubernetes.io/projected/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-token\") pod \"4a521083-7066-42d2-8997-7e97d4a7ef3e\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " Oct 31 20:05:11.012066 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.011611 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-podinfo\") pod \"4a521083-7066-42d2-8997-7e97d4a7ef3e\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " Oct 31 20:05:11.012066 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.011655 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-workload-socket" pod "4a521083-7066-42d2-8997-7e97d4a7ef3e" (UID: "4a521083-7066-42d2-8997-7e97d4a7ef3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:11.012066 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.011668 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-data" pod "4a521083-7066-42d2-8997-7e97d4a7ef3e" (UID: "4a521083-7066-42d2-8997-7e97d4a7ef3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:11.012066 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.011822 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-workload-certs\") pod \"4a521083-7066-42d2-8997-7e97d4a7ef3e\" (UID: \"4a521083-7066-42d2-8997-7e97d4a7ef3e\") " Oct 31 20:05:11.012066 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.011961 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a521083-7066-42d2-8997-7e97d4a7ef3e-istiod-ca-cert" pod "4a521083-7066-42d2-8997-7e97d4a7ef3e" (UID: "4a521083-7066-42d2-8997-7e97d4a7ef3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:05:11.012328 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.012119 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-credential-socket" pod "4a521083-7066-42d2-8997-7e97d4a7ef3e" (UID: "4a521083-7066-42d2-8997-7e97d4a7ef3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:11.012368 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.012324 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-workload-certs" pod "4a521083-7066-42d2-8997-7e97d4a7ef3e" (UID: "4a521083-7066-42d2-8997-7e97d4a7ef3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:11.012489 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.012467 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:11.012540 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.012495 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:11.012540 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.012510 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a521083-7066-42d2-8997-7e97d4a7ef3e-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:11.012540 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.012525 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:11.012690 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.012541 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:11.013789 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.013761 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-envoy" pod "4a521083-7066-42d2-8997-7e97d4a7ef3e" (UID: "4a521083-7066-42d2-8997-7e97d4a7ef3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:11.013953 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.013930 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-podinfo" pod "4a521083-7066-42d2-8997-7e97d4a7ef3e" (UID: "4a521083-7066-42d2-8997-7e97d4a7ef3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:05:11.014016 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.013948 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a521083-7066-42d2-8997-7e97d4a7ef3e-kube-api-access-4x4wr" pod "4a521083-7066-42d2-8997-7e97d4a7ef3e" (UID: "4a521083-7066-42d2-8997-7e97d4a7ef3e"). InnerVolumeSpecName "kube-api-access-4x4wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:11.014238 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.014219 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-token" pod "4a521083-7066-42d2-8997-7e97d4a7ef3e" (UID: "4a521083-7066-42d2-8997-7e97d4a7ef3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:11.114263 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.114222 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:11.114263 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.114257 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x4wr\" (UniqueName: \"kubernetes.io/projected/4a521083-7066-42d2-8997-7e97d4a7ef3e-kube-api-access-4x4wr\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:11.114263 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.114269 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:11.114492 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.114278 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a521083-7066-42d2-8997-7e97d4a7ef3e-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:11.708151 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.708057 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" Oct 31 20:05:11.708151 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.708087 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf" event={"ID":"4a521083-7066-42d2-8997-7e97d4a7ef3e","Type":"ContainerDied","Data":"5d012232d45e12a4ad65ae1d680234f057f3a5e896687e249cdef5869bb9fa9a"} Oct 31 20:05:11.708151 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.708152 4910 scope.go:122] "RemoveContainer" containerID="998030f55119599ed804de17e403d16183f6e93439c046fd91588ed5a7148824" Oct 31 20:05:11.709858 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.709825 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" Oct 31 20:05:11.710040 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.709858 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7" event={"ID":"64efa0c3-abc5-4253-8f01-cb9954e33dc6","Type":"ContainerDied","Data":"5207a1e72f6aedc629ffa66ffb3a5ec89ebf0c9cf983324cebc6f0b086463712"} Oct 31 20:05:11.723711 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.723687 4910 scope.go:122] "RemoveContainer" containerID="e722a46bfbd56c97023369be32ecbd1a3382095ec2e5c1d9c950aff77459dc7d" Oct 31 20:05:11.761368 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.761339 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf"] Oct 31 20:05:11.767866 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.767838 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4pmnf"] Oct 31 20:05:11.780084 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.780057 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7"] Oct 31 20:05:11.786881 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:11.786852 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-kd7l7"] Oct 31 20:05:12.715046 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:05:12.715006 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:05:13.755456 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:13.755418 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a521083-7066-42d2-8997-7e97d4a7ef3e" path="/var/lib/kubelet/pods/4a521083-7066-42d2-8997-7e97d4a7ef3e/volumes" Oct 31 20:05:13.755979 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:13.755931 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64efa0c3-abc5-4253-8f01-cb9954e33dc6" path="/var/lib/kubelet/pods/64efa0c3-abc5-4253-8f01-cb9954e33dc6/volumes" Oct 31 20:05:15.032541 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:15.032508 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16883d53-tvt6g"] Oct 31 20:05:15.036133 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:15.036105 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16883d53-tvt6g"] Oct 31 20:05:15.756588 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:15.756552 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52f21d22-e27f-45eb-ae8f-9d971b0baa9e" path="/var/lib/kubelet/pods/52f21d22-e27f-45eb-ae8f-9d971b0baa9e/volumes" Oct 31 20:05:16.171472 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.171439 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs"] Oct 31 20:05:16.172189 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.172157 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" podUID="ce05955e-1925-4469-978f-29f94fc3d6f0" containerName="istio-proxy" containerID="cri-o://5a1bb571db833edf9c804ba2527b8fb1e84361d13c52cdd186ee06d716312cb7" gracePeriod=30 Oct 31 20:05:16.175864 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.175837 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf"] Oct 31 20:05:16.176399 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.176040 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" podUID="59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13" containerName="istio-proxy" containerID="cri-o://377cb0825ffe594216103119e3715f89a5b6cd0405aaba7c79e71a297479a351" gracePeriod=30 Oct 31 20:05:16.705182 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.705146 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w"] Oct 31 20:05:16.705800 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.705784 4910 cpu_manager.go:414] "containerMap: removing container" podUID="4a521083-7066-42d2-8997-7e97d4a7ef3e" containerName="istio-proxy" Oct 31 20:05:16.705957 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.705802 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a521083-7066-42d2-8997-7e97d4a7ef3e" containerName="istio-proxy" Oct 31 20:05:16.705957 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.705816 4910 cpu_manager.go:414] "containerMap: removing container" podUID="64efa0c3-abc5-4253-8f01-cb9954e33dc6" containerName="istio-proxy" Oct 31 20:05:16.705957 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.705821 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64efa0c3-abc5-4253-8f01-cb9954e33dc6" containerName="istio-proxy" Oct 31 20:05:16.705957 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.705830 4910 cpu_manager.go:414] "containerMap: removing container" podUID="a1234fff-2cf6-40b7-ab83-bd37f5a75369" containerName="revoke-keys" Oct 31 20:05:16.705957 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.705835 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1234fff-2cf6-40b7-ab83-bd37f5a75369" containerName="revoke-keys" Oct 31 20:05:16.706133 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.705978 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a521083-7066-42d2-8997-7e97d4a7ef3e" containerName="istio-proxy" Oct 31 20:05:16.706133 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.705986 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="64efa0c3-abc5-4253-8f01-cb9954e33dc6" containerName="istio-proxy" Oct 31 20:05:16.706133 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.705993 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1234fff-2cf6-40b7-ab83-bd37f5a75369" containerName="revoke-keys" Oct 31 20:05:16.711023 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.711005 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w" Oct 31 20:05:16.716425 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.716399 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:05:16.716573 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.716509 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 20:05:16.720558 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.720534 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w"] Oct 31 20:05:16.833636 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.833587 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee6f79c-82ab-4c26-abb9-038ad03a9dc0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w\" (UID: \"eee6f79c-82ab-4c26-abb9-038ad03a9dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w" Oct 31 20:05:16.833856 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.833716 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdz5g\" (UniqueName: \"kubernetes.io/projected/eee6f79c-82ab-4c26-abb9-038ad03a9dc0-kube-api-access-tdz5g\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w\" (UID: \"eee6f79c-82ab-4c26-abb9-038ad03a9dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w" Oct 31 20:05:16.833945 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.833888 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee6f79c-82ab-4c26-abb9-038ad03a9dc0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w\" (UID: \"eee6f79c-82ab-4c26-abb9-038ad03a9dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w" Oct 31 20:05:16.935989 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.935935 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee6f79c-82ab-4c26-abb9-038ad03a9dc0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w\" (UID: \"eee6f79c-82ab-4c26-abb9-038ad03a9dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w" Oct 31 20:05:16.936180 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.936139 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee6f79c-82ab-4c26-abb9-038ad03a9dc0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w\" (UID: \"eee6f79c-82ab-4c26-abb9-038ad03a9dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w" Oct 31 20:05:16.936253 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.936227 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdz5g\" (UniqueName: \"kubernetes.io/projected/eee6f79c-82ab-4c26-abb9-038ad03a9dc0-kube-api-access-tdz5g\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w\" (UID: \"eee6f79c-82ab-4c26-abb9-038ad03a9dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w" Oct 31 20:05:16.936671 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.936639 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee6f79c-82ab-4c26-abb9-038ad03a9dc0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w\" (UID: \"eee6f79c-82ab-4c26-abb9-038ad03a9dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w" Oct 31 20:05:16.939000 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.938971 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee6f79c-82ab-4c26-abb9-038ad03a9dc0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w\" (UID: \"eee6f79c-82ab-4c26-abb9-038ad03a9dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w" Oct 31 20:05:16.948892 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:16.948860 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdz5g\" (UniqueName: \"kubernetes.io/projected/eee6f79c-82ab-4c26-abb9-038ad03a9dc0-kube-api-access-tdz5g\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w\" (UID: \"eee6f79c-82ab-4c26-abb9-038ad03a9dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w" Oct 31 20:05:17.036798 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:17.036711 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w" Oct 31 20:05:17.246833 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:17.246792 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:05:17.247267 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:17.246865 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" podUID="59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:05:17.253897 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:17.253862 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:05:17.254044 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:17.253950 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" podUID="ce05955e-1925-4469-978f-29f94fc3d6f0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:05:17.322695 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:17.322662 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w"] Oct 31 20:05:17.323652 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:05:17.323621 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee6f79c_82ab_4c26_abb9_038ad03a9dc0.slice/crio-4c90411fc791944c3923d936f36f29ec34d7ee43ea4ae7f454b3feb44bf8fa9f WatchSource:0}: Error finding container 4c90411fc791944c3923d936f36f29ec34d7ee43ea4ae7f454b3feb44bf8fa9f: Status 404 returned error can't find the container with id 4c90411fc791944c3923d936f36f29ec34d7ee43ea4ae7f454b3feb44bf8fa9f Oct 31 20:05:17.771633 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:17.771594 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w" event={"ID":"eee6f79c-82ab-4c26-abb9-038ad03a9dc0","Type":"ContainerStarted","Data":"6f2f35fcbffe7fc149be7b4d6989bbb25b404ec32a6d10891bc94c2697bd625f"} Oct 31 20:05:17.772530 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:17.771655 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w" event={"ID":"eee6f79c-82ab-4c26-abb9-038ad03a9dc0","Type":"ContainerStarted","Data":"4c90411fc791944c3923d936f36f29ec34d7ee43ea4ae7f454b3feb44bf8fa9f"} Oct 31 20:05:17.805688 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:17.805536 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w" podStartSLOduration=1.805514024 podStartE2EDuration="1.805514024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:17.790979615 +0000 UTC m=+2752.153799987" watchObservedRunningTime="2026-10-31 20:05:17.805514024 +0000 UTC m=+2752.168334400" Oct 31 20:05:18.616514 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:18.616471 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc"] Oct 31 20:05:18.617019 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:18.616777 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" podUID="922b063f-4aee-4ed2-9355-3cd5222862df" containerName="istio-proxy" containerID="cri-o://ea1885212f37c461eb2ee862b79f1b42373c21f2cf365a087849a56c55ac8550" gracePeriod=30 Oct 31 20:05:18.818395 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:18.818356 4910 generic.go:356] "Generic (PLEG): container finished" podID="eee6f79c-82ab-4c26-abb9-038ad03a9dc0" containerID="6f2f35fcbffe7fc149be7b4d6989bbb25b404ec32a6d10891bc94c2697bd625f" exitCode=0 Oct 31 20:05:18.818695 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:18.818671 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w" event={"ID":"eee6f79c-82ab-4c26-abb9-038ad03a9dc0","Type":"ContainerDied","Data":"6f2f35fcbffe7fc149be7b4d6989bbb25b404ec32a6d10891bc94c2697bd625f"} Oct 31 20:05:20.004567 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:20.004536 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w" Oct 31 20:05:20.106798 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:20.106717 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eee6f79c-82ab-4c26-abb9-038ad03a9dc0-kube-api-access-tdz5g\" (UniqueName: \"kubernetes.io/projected/eee6f79c-82ab-4c26-abb9-038ad03a9dc0-kube-api-access-tdz5g\") pod \"eee6f79c-82ab-4c26-abb9-038ad03a9dc0\" (UID: \"eee6f79c-82ab-4c26-abb9-038ad03a9dc0\") " Oct 31 20:05:20.106982 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:20.106850 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eee6f79c-82ab-4c26-abb9-038ad03a9dc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee6f79c-82ab-4c26-abb9-038ad03a9dc0-maas-api-openshift-ca\") pod \"eee6f79c-82ab-4c26-abb9-038ad03a9dc0\" (UID: \"eee6f79c-82ab-4c26-abb9-038ad03a9dc0\") " Oct 31 20:05:20.106982 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:20.106885 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eee6f79c-82ab-4c26-abb9-038ad03a9dc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee6f79c-82ab-4c26-abb9-038ad03a9dc0-maas-api-xks-ca\") pod \"eee6f79c-82ab-4c26-abb9-038ad03a9dc0\" (UID: \"eee6f79c-82ab-4c26-abb9-038ad03a9dc0\") " Oct 31 20:05:20.107243 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:20.107220 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eee6f79c-82ab-4c26-abb9-038ad03a9dc0-maas-api-openshift-ca" pod "eee6f79c-82ab-4c26-abb9-038ad03a9dc0" (UID: "eee6f79c-82ab-4c26-abb9-038ad03a9dc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:05:20.107548 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:20.107532 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eee6f79c-82ab-4c26-abb9-038ad03a9dc0-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:20.109122 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:20.109083 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eee6f79c-82ab-4c26-abb9-038ad03a9dc0-maas-api-xks-ca" pod "eee6f79c-82ab-4c26-abb9-038ad03a9dc0" (UID: "eee6f79c-82ab-4c26-abb9-038ad03a9dc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:05:20.109252 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:20.109165 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee6f79c-82ab-4c26-abb9-038ad03a9dc0-kube-api-access-tdz5g" pod "eee6f79c-82ab-4c26-abb9-038ad03a9dc0" (UID: "eee6f79c-82ab-4c26-abb9-038ad03a9dc0"). InnerVolumeSpecName "kube-api-access-tdz5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:20.209474 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:20.209431 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eee6f79c-82ab-4c26-abb9-038ad03a9dc0-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:20.209474 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:20.209466 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdz5g\" (UniqueName: \"kubernetes.io/projected/eee6f79c-82ab-4c26-abb9-038ad03a9dc0-kube-api-access-tdz5g\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:20.838877 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:20.838843 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w" Oct 31 20:05:20.839200 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:20.838872 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w" event={"ID":"eee6f79c-82ab-4c26-abb9-038ad03a9dc0","Type":"ContainerDied","Data":"4c90411fc791944c3923d936f36f29ec34d7ee43ea4ae7f454b3feb44bf8fa9f"} Oct 31 20:05:20.839200 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:20.838964 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c90411fc791944c3923d936f36f29ec34d7ee43ea4ae7f454b3feb44bf8fa9f" Oct 31 20:05:21.136194 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.136103 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d613866-7sb6w"] Oct 31 20:05:21.137116 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.137090 4910 cpu_manager.go:414] "containerMap: removing container" podUID="64efa0c3-abc5-4253-8f01-cb9954e33dc6" containerName="istio-proxy" Oct 31 20:05:21.137234 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.137117 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64efa0c3-abc5-4253-8f01-cb9954e33dc6" containerName="istio-proxy" Oct 31 20:05:21.137234 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.137141 4910 cpu_manager.go:414] "containerMap: removing container" podUID="eee6f79c-82ab-4c26-abb9-038ad03a9dc0" containerName="revoke-keys" Oct 31 20:05:21.137234 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.137149 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eee6f79c-82ab-4c26-abb9-038ad03a9dc0" containerName="revoke-keys" Oct 31 20:05:21.137347 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.137334 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="64efa0c3-abc5-4253-8f01-cb9954e33dc6" containerName="istio-proxy" Oct 31 20:05:21.137383 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.137350 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="eee6f79c-82ab-4c26-abb9-038ad03a9dc0" containerName="revoke-keys" Oct 31 20:05:21.141872 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.141850 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d613866-7sb6w" Oct 31 20:05:21.151800 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.151776 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 20:05:21.152989 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.152036 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:05:21.158366 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.158341 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d613866-7sb6w"] Oct 31 20:05:21.174576 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.174536 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1b2c5fc-n4b69"] Oct 31 20:05:21.188816 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.186903 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b2c5fc-n4b69" Oct 31 20:05:21.196467 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.196434 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1b2c5fc-n4b69"] Oct 31 20:05:21.231756 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.231719 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/698675aa-5484-476d-9b7a-e677c312dcc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1b2c5fc-n4b69\" (UID: \"698675aa-5484-476d-9b7a-e677c312dcc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b2c5fc-n4b69" Oct 31 20:05:21.231925 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.231807 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/698675aa-5484-476d-9b7a-e677c312dcc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1b2c5fc-n4b69\" (UID: \"698675aa-5484-476d-9b7a-e677c312dcc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b2c5fc-n4b69" Oct 31 20:05:21.231925 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.231855 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad7d564e-a00c-4722-9849-dcbd5de16477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d613866-7sb6w\" (UID: \"ad7d564e-a00c-4722-9849-dcbd5de16477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d613866-7sb6w" Oct 31 20:05:21.231925 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.231883 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq6w9\" (UniqueName: \"kubernetes.io/projected/698675aa-5484-476d-9b7a-e677c312dcc3-kube-api-access-vq6w9\") pod \"maas-api-revoke-sub-d1b2c5fc-n4b69\" (UID: \"698675aa-5484-476d-9b7a-e677c312dcc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b2c5fc-n4b69" Oct 31 20:05:21.232132 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.231961 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4422\" (UniqueName: \"kubernetes.io/projected/ad7d564e-a00c-4722-9849-dcbd5de16477-kube-api-access-x4422\") pod \"maas-api-revoke-sub-2d613866-7sb6w\" (UID: \"ad7d564e-a00c-4722-9849-dcbd5de16477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d613866-7sb6w" Oct 31 20:05:21.232195 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.232182 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad7d564e-a00c-4722-9849-dcbd5de16477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d613866-7sb6w\" (UID: \"ad7d564e-a00c-4722-9849-dcbd5de16477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d613866-7sb6w" Oct 31 20:05:21.346975 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.333530 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad7d564e-a00c-4722-9849-dcbd5de16477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d613866-7sb6w\" (UID: \"ad7d564e-a00c-4722-9849-dcbd5de16477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d613866-7sb6w" Oct 31 20:05:21.346975 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.333634 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/698675aa-5484-476d-9b7a-e677c312dcc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1b2c5fc-n4b69\" (UID: \"698675aa-5484-476d-9b7a-e677c312dcc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b2c5fc-n4b69" Oct 31 20:05:21.346975 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.333677 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/698675aa-5484-476d-9b7a-e677c312dcc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1b2c5fc-n4b69\" (UID: \"698675aa-5484-476d-9b7a-e677c312dcc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b2c5fc-n4b69" Oct 31 20:05:21.346975 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.333705 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad7d564e-a00c-4722-9849-dcbd5de16477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d613866-7sb6w\" (UID: \"ad7d564e-a00c-4722-9849-dcbd5de16477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d613866-7sb6w" Oct 31 20:05:21.346975 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.333724 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq6w9\" (UniqueName: \"kubernetes.io/projected/698675aa-5484-476d-9b7a-e677c312dcc3-kube-api-access-vq6w9\") pod \"maas-api-revoke-sub-d1b2c5fc-n4b69\" (UID: \"698675aa-5484-476d-9b7a-e677c312dcc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b2c5fc-n4b69" Oct 31 20:05:21.346975 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.333760 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4422\" (UniqueName: \"kubernetes.io/projected/ad7d564e-a00c-4722-9849-dcbd5de16477-kube-api-access-x4422\") pod \"maas-api-revoke-sub-2d613866-7sb6w\" (UID: \"ad7d564e-a00c-4722-9849-dcbd5de16477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d613866-7sb6w" Oct 31 20:05:21.346975 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.335334 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad7d564e-a00c-4722-9849-dcbd5de16477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d613866-7sb6w\" (UID: \"ad7d564e-a00c-4722-9849-dcbd5de16477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d613866-7sb6w" Oct 31 20:05:21.346975 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.336014 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/698675aa-5484-476d-9b7a-e677c312dcc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1b2c5fc-n4b69\" (UID: \"698675aa-5484-476d-9b7a-e677c312dcc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b2c5fc-n4b69" Oct 31 20:05:21.346975 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.337025 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad7d564e-a00c-4722-9849-dcbd5de16477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d613866-7sb6w\" (UID: \"ad7d564e-a00c-4722-9849-dcbd5de16477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d613866-7sb6w" Oct 31 20:05:21.346975 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.337466 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/698675aa-5484-476d-9b7a-e677c312dcc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1b2c5fc-n4b69\" (UID: \"698675aa-5484-476d-9b7a-e677c312dcc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b2c5fc-n4b69" Oct 31 20:05:21.355627 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.355591 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq6w9\" (UniqueName: \"kubernetes.io/projected/698675aa-5484-476d-9b7a-e677c312dcc3-kube-api-access-vq6w9\") pod \"maas-api-revoke-sub-d1b2c5fc-n4b69\" (UID: \"698675aa-5484-476d-9b7a-e677c312dcc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b2c5fc-n4b69" Oct 31 20:05:21.357716 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.357683 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4422\" (UniqueName: \"kubernetes.io/projected/ad7d564e-a00c-4722-9849-dcbd5de16477-kube-api-access-x4422\") pod \"maas-api-revoke-sub-2d613866-7sb6w\" (UID: \"ad7d564e-a00c-4722-9849-dcbd5de16477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d613866-7sb6w" Oct 31 20:05:21.400078 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.400014 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:05:21.434970 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.434926 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istiod-ca-cert\") pod \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " Oct 31 20:05:21.435157 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.434983 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-envoy\") pod \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " Oct 31 20:05:21.435157 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.435010 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-credential-socket\") pod \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " Oct 31 20:05:21.435157 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.435083 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-podinfo\") pod \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " Oct 31 20:05:21.435157 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.435137 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-data\") pod \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " Oct 31 20:05:21.435588 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.435190 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-workload-socket\") pod \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " Oct 31 20:05:21.435588 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.435244 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-workload-certs\") pod \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " Oct 31 20:05:21.435588 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.435312 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-token\" (UniqueName: \"kubernetes.io/projected/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-token\") pod \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " Oct 31 20:05:21.435588 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.435421 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-kube-api-access-mlsph\" (UniqueName: \"kubernetes.io/projected/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-kube-api-access-mlsph\") pod \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\" (UID: \"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13\") " Oct 31 20:05:21.436380 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.436161 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-data" pod "59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13" (UID: "59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:21.436380 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.436244 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-workload-socket" pod "59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13" (UID: "59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:21.436666 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.436641 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-credential-socket" pod "59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13" (UID: "59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:21.436729 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.436658 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-workload-certs" pod "59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13" (UID: "59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:21.436786 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.436752 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istiod-ca-cert" pod "59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13" (UID: "59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:05:21.439124 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.439084 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-envoy" pod "59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13" (UID: "59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:21.439384 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.439358 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-podinfo" pod "59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13" (UID: "59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:05:21.441642 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.441607 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-kube-api-access-mlsph" pod "59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13" (UID: "59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13"). InnerVolumeSpecName "kube-api-access-mlsph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:21.460896 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.460870 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d613866-7sb6w" Oct 31 20:05:21.463465 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.463438 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-token" pod "59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13" (UID: "59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:21.540728 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.538042 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:21.540728 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.538083 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:21.540728 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.538101 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:21.540728 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.538116 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:21.540728 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.538131 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:21.540728 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.538147 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:21.540728 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.538160 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:21.540728 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.538173 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:21.540728 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.538188 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlsph\" (UniqueName: \"kubernetes.io/projected/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13-kube-api-access-mlsph\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:21.612810 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.612776 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b2c5fc-n4b69" Oct 31 20:05:21.810062 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.810029 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d613866-7sb6w"] Oct 31 20:05:21.810447 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:05:21.810420 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad7d564e_a00c_4722_9849_dcbd5de16477.slice/crio-3bc1dc28fd575a7978f5e00c519802b2140a820aca2fcdea99c4e395619d03ed WatchSource:0}: Error finding container 3bc1dc28fd575a7978f5e00c519802b2140a820aca2fcdea99c4e395619d03ed: Status 404 returned error can't find the container with id 3bc1dc28fd575a7978f5e00c519802b2140a820aca2fcdea99c4e395619d03ed Oct 31 20:05:21.866134 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.866091 4910 generic.go:356] "Generic (PLEG): container finished" podID="59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13" containerID="377cb0825ffe594216103119e3715f89a5b6cd0405aaba7c79e71a297479a351" exitCode=0 Oct 31 20:05:21.866553 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.866531 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" event={"ID":"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13","Type":"ContainerDied","Data":"377cb0825ffe594216103119e3715f89a5b6cd0405aaba7c79e71a297479a351"} Oct 31 20:05:21.866749 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.866732 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" event={"ID":"59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13","Type":"ContainerDied","Data":"839b548b30793416af7eb70514ecf5cde97864a8e86320a24622a7b25dede4a6"} Oct 31 20:05:21.866891 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.866876 4910 scope.go:122] "RemoveContainer" containerID="377cb0825ffe594216103119e3715f89a5b6cd0405aaba7c79e71a297479a351" Oct 31 20:05:21.867062 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.867041 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf" Oct 31 20:05:21.869883 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.869703 4910 generic.go:356] "Generic (PLEG): container finished" podID="ce05955e-1925-4469-978f-29f94fc3d6f0" containerID="5a1bb571db833edf9c804ba2527b8fb1e84361d13c52cdd186ee06d716312cb7" exitCode=0 Oct 31 20:05:21.869883 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.869830 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" event={"ID":"ce05955e-1925-4469-978f-29f94fc3d6f0","Type":"ContainerDied","Data":"5a1bb571db833edf9c804ba2527b8fb1e84361d13c52cdd186ee06d716312cb7"} Oct 31 20:05:21.871732 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.871709 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d613866-7sb6w" event={"ID":"ad7d564e-a00c-4722-9849-dcbd5de16477","Type":"ContainerStarted","Data":"3bc1dc28fd575a7978f5e00c519802b2140a820aca2fcdea99c4e395619d03ed"} Oct 31 20:05:21.885359 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.884813 4910 scope.go:122] "RemoveContainer" containerID="377cb0825ffe594216103119e3715f89a5b6cd0405aaba7c79e71a297479a351" Oct 31 20:05:21.885359 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.884695 4910 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:05:21.885584 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.885386 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" podUID="922b063f-4aee-4ed2-9355-3cd5222862df" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:05:21.885584 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:05:21.885536 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"377cb0825ffe594216103119e3715f89a5b6cd0405aaba7c79e71a297479a351\": container with ID starting with 377cb0825ffe594216103119e3715f89a5b6cd0405aaba7c79e71a297479a351 not found: ID does not exist" containerID="377cb0825ffe594216103119e3715f89a5b6cd0405aaba7c79e71a297479a351" Oct 31 20:05:21.885715 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.885599 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377cb0825ffe594216103119e3715f89a5b6cd0405aaba7c79e71a297479a351"} err="failed to get container status \"377cb0825ffe594216103119e3715f89a5b6cd0405aaba7c79e71a297479a351\": rpc error: code = NotFound desc = could not find container \"377cb0825ffe594216103119e3715f89a5b6cd0405aaba7c79e71a297479a351\": container with ID starting with 377cb0825ffe594216103119e3715f89a5b6cd0405aaba7c79e71a297479a351 not found: ID does not exist" Oct 31 20:05:21.895202 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.895175 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf"] Oct 31 20:05:21.900006 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.899983 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 20:05:21.902482 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.902457 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-s5hlf"] Oct 31 20:05:21.918893 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:05:21.918867 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698675aa_5484_476d_9b7a_e677c312dcc3.slice/crio-2cc9ce129238fee154fbe85173b8959bb69ffa504404757b30524dfaff488c5f WatchSource:0}: Error finding container 2cc9ce129238fee154fbe85173b8959bb69ffa504404757b30524dfaff488c5f: Status 404 returned error can't find the container with id 2cc9ce129238fee154fbe85173b8959bb69ffa504404757b30524dfaff488c5f Oct 31 20:05:21.922530 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.922507 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1b2c5fc-n4b69"] Oct 31 20:05:21.952303 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.952202 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-envoy\") pod \"ce05955e-1925-4469-978f-29f94fc3d6f0\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " Oct 31 20:05:21.952484 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.952387 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce05955e-1925-4469-978f-29f94fc3d6f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce05955e-1925-4469-978f-29f94fc3d6f0-istiod-ca-cert\") pod \"ce05955e-1925-4469-978f-29f94fc3d6f0\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " Oct 31 20:05:21.952853 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.952822 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce05955e-1925-4469-978f-29f94fc3d6f0-istiod-ca-cert" pod "ce05955e-1925-4469-978f-29f94fc3d6f0" (UID: "ce05955e-1925-4469-978f-29f94fc3d6f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:05:21.953003 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.952952 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-podinfo\") pod \"ce05955e-1925-4469-978f-29f94fc3d6f0\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " Oct 31 20:05:21.953324 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.953300 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-data" pod "ce05955e-1925-4469-978f-29f94fc3d6f0" (UID: "ce05955e-1925-4469-978f-29f94fc3d6f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:21.953600 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.953577 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-data\") pod \"ce05955e-1925-4469-978f-29f94fc3d6f0\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " Oct 31 20:05:21.953858 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.953841 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-token\" (UniqueName: \"kubernetes.io/projected/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-token\") pod \"ce05955e-1925-4469-978f-29f94fc3d6f0\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " Oct 31 20:05:21.954374 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.954353 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce05955e-1925-4469-978f-29f94fc3d6f0-kube-api-access-ctvzx\" (UniqueName: \"kubernetes.io/projected/ce05955e-1925-4469-978f-29f94fc3d6f0-kube-api-access-ctvzx\") pod \"ce05955e-1925-4469-978f-29f94fc3d6f0\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " Oct 31 20:05:21.954970 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.954952 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-workload-certs\") pod \"ce05955e-1925-4469-978f-29f94fc3d6f0\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " Oct 31 20:05:21.955700 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.955413 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-workload-socket\") pod \"ce05955e-1925-4469-978f-29f94fc3d6f0\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " Oct 31 20:05:21.955904 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.955354 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-workload-certs" pod "ce05955e-1925-4469-978f-29f94fc3d6f0" (UID: "ce05955e-1925-4469-978f-29f94fc3d6f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:21.955904 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.955638 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-workload-socket" pod "ce05955e-1925-4469-978f-29f94fc3d6f0" (UID: "ce05955e-1925-4469-978f-29f94fc3d6f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:21.956161 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.956141 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-credential-socket\") pod \"ce05955e-1925-4469-978f-29f94fc3d6f0\" (UID: \"ce05955e-1925-4469-978f-29f94fc3d6f0\") " Oct 31 20:05:21.956570 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.956462 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-credential-socket" pod "ce05955e-1925-4469-978f-29f94fc3d6f0" (UID: "ce05955e-1925-4469-978f-29f94fc3d6f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:21.957298 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.957275 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce05955e-1925-4469-978f-29f94fc3d6f0-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:21.957432 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.957415 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:21.957533 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.957520 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:21.957628 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.957618 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:21.957690 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:21.957683 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:22.019609 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:22.019566 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-podinfo" pod "ce05955e-1925-4469-978f-29f94fc3d6f0" (UID: "ce05955e-1925-4469-978f-29f94fc3d6f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:05:22.019779 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:22.019603 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-envoy" pod "ce05955e-1925-4469-978f-29f94fc3d6f0" (UID: "ce05955e-1925-4469-978f-29f94fc3d6f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:22.020029 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:22.020005 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-token" pod "ce05955e-1925-4469-978f-29f94fc3d6f0" (UID: "ce05955e-1925-4469-978f-29f94fc3d6f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:22.020294 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:22.020274 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce05955e-1925-4469-978f-29f94fc3d6f0-kube-api-access-ctvzx" pod "ce05955e-1925-4469-978f-29f94fc3d6f0" (UID: "ce05955e-1925-4469-978f-29f94fc3d6f0"). InnerVolumeSpecName "kube-api-access-ctvzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:22.059753 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:22.059712 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:22.059753 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:22.059755 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:22.059990 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:22.059770 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce05955e-1925-4469-978f-29f94fc3d6f0-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:22.059990 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:22.059785 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctvzx\" (UniqueName: \"kubernetes.io/projected/ce05955e-1925-4469-978f-29f94fc3d6f0-kube-api-access-ctvzx\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:22.896274 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:22.896236 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b2c5fc-n4b69" event={"ID":"698675aa-5484-476d-9b7a-e677c312dcc3","Type":"ContainerStarted","Data":"91687e4d221e5ddfc0f0d3139b0752a7d173b4c49ba85ad39d8a321782491bea"} Oct 31 20:05:22.896736 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:22.896298 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b2c5fc-n4b69" event={"ID":"698675aa-5484-476d-9b7a-e677c312dcc3","Type":"ContainerStarted","Data":"2cc9ce129238fee154fbe85173b8959bb69ffa504404757b30524dfaff488c5f"} Oct 31 20:05:22.897810 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:22.897789 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" Oct 31 20:05:22.897933 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:22.897825 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs" event={"ID":"ce05955e-1925-4469-978f-29f94fc3d6f0","Type":"ContainerDied","Data":"d4e813422d5d9f357077f03afb88227da08ccdfe08ed1c5913619a2b551c215b"} Oct 31 20:05:22.897933 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:22.897884 4910 scope.go:122] "RemoveContainer" containerID="5a1bb571db833edf9c804ba2527b8fb1e84361d13c52cdd186ee06d716312cb7" Oct 31 20:05:22.899793 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:22.899764 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d613866-7sb6w" event={"ID":"ad7d564e-a00c-4722-9849-dcbd5de16477","Type":"ContainerStarted","Data":"6c327f9d547fcc0ad8e3b455630f403134df2d1a3e7dbe56f115905cda2ed460"} Oct 31 20:05:22.918987 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:22.918896 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b2c5fc-n4b69" podStartSLOduration=1.9188824850000001 podStartE2EDuration="1.918882485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:22.917720575 +0000 UTC m=+2757.280540967" watchObservedRunningTime="2026-10-31 20:05:22.918882485 +0000 UTC m=+2757.281702857" Oct 31 20:05:22.938691 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:22.938634 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d613866-7sb6w" podStartSLOduration=1.938615032 podStartE2EDuration="1.938615032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:22.937344713 +0000 UTC m=+2757.300165105" watchObservedRunningTime="2026-10-31 20:05:22.938615032 +0000 UTC m=+2757.301435404" Oct 31 20:05:22.963015 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:22.962985 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs"] Oct 31 20:05:22.968968 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:22.968941 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-rcvzs"] Oct 31 20:05:23.786768 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:23.786726 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13" path="/var/lib/kubelet/pods/59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13/volumes" Oct 31 20:05:23.799505 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:23.799469 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce05955e-1925-4469-978f-29f94fc3d6f0" path="/var/lib/kubelet/pods/ce05955e-1925-4469-978f-29f94fc3d6f0/volumes" Oct 31 20:05:23.920036 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:23.919994 4910 generic.go:356] "Generic (PLEG): container finished" podID="922b063f-4aee-4ed2-9355-3cd5222862df" containerID="ea1885212f37c461eb2ee862b79f1b42373c21f2cf365a087849a56c55ac8550" exitCode=0 Oct 31 20:05:23.920462 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:23.920072 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" event={"ID":"922b063f-4aee-4ed2-9355-3cd5222862df","Type":"ContainerDied","Data":"ea1885212f37c461eb2ee862b79f1b42373c21f2cf365a087849a56c55ac8550"} Oct 31 20:05:23.920462 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:23.920147 4910 scope.go:122] "RemoveContainer" containerID="f40141a0f0edd3d9dbf89b3b2852783ef462ab01f27fe56d3f41f5db6e47b93a" Oct 31 20:05:23.924258 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:23.924109 4910 generic.go:356] "Generic (PLEG): container finished" podID="698675aa-5484-476d-9b7a-e677c312dcc3" containerID="91687e4d221e5ddfc0f0d3139b0752a7d173b4c49ba85ad39d8a321782491bea" exitCode=0 Oct 31 20:05:23.924258 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:23.924180 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b2c5fc-n4b69" event={"ID":"698675aa-5484-476d-9b7a-e677c312dcc3","Type":"ContainerDied","Data":"91687e4d221e5ddfc0f0d3139b0752a7d173b4c49ba85ad39d8a321782491bea"} Oct 31 20:05:23.928209 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:23.927940 4910 generic.go:356] "Generic (PLEG): container finished" podID="ad7d564e-a00c-4722-9849-dcbd5de16477" containerID="6c327f9d547fcc0ad8e3b455630f403134df2d1a3e7dbe56f115905cda2ed460" exitCode=0 Oct 31 20:05:23.928209 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:23.927991 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d613866-7sb6w" event={"ID":"ad7d564e-a00c-4722-9849-dcbd5de16477","Type":"ContainerDied","Data":"6c327f9d547fcc0ad8e3b455630f403134df2d1a3e7dbe56f115905cda2ed460"} Oct 31 20:05:24.224643 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.224326 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 20:05:24.327955 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.327922 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-workload-certs\") pod \"922b063f-4aee-4ed2-9355-3cd5222862df\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " Oct 31 20:05:24.328137 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.327982 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-istio-data\") pod \"922b063f-4aee-4ed2-9355-3cd5222862df\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " Oct 31 20:05:24.328137 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.328020 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922b063f-4aee-4ed2-9355-3cd5222862df-istio-token\" (UniqueName: \"kubernetes.io/projected/922b063f-4aee-4ed2-9355-3cd5222862df-istio-token\") pod \"922b063f-4aee-4ed2-9355-3cd5222862df\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " Oct 31 20:05:24.328137 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.328110 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-credential-socket\") pod \"922b063f-4aee-4ed2-9355-3cd5222862df\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " Oct 31 20:05:24.328333 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.328175 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-istio-envoy\") pod \"922b063f-4aee-4ed2-9355-3cd5222862df\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " Oct 31 20:05:24.328333 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.328234 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-workload-certs" pod "922b063f-4aee-4ed2-9355-3cd5222862df" (UID: "922b063f-4aee-4ed2-9355-3cd5222862df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:24.328333 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.328259 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-istio-data" pod "922b063f-4aee-4ed2-9355-3cd5222862df" (UID: "922b063f-4aee-4ed2-9355-3cd5222862df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:24.328333 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.328264 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922b063f-4aee-4ed2-9355-3cd5222862df-kube-api-access-97z6l\" (UniqueName: \"kubernetes.io/projected/922b063f-4aee-4ed2-9355-3cd5222862df-kube-api-access-97z6l\") pod \"922b063f-4aee-4ed2-9355-3cd5222862df\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " Oct 31 20:05:24.328553 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.328337 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-workload-socket\") pod \"922b063f-4aee-4ed2-9355-3cd5222862df\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " Oct 31 20:05:24.328553 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.328402 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/922b063f-4aee-4ed2-9355-3cd5222862df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922b063f-4aee-4ed2-9355-3cd5222862df-istiod-ca-cert\") pod \"922b063f-4aee-4ed2-9355-3cd5222862df\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " Oct 31 20:05:24.328553 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.328421 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/922b063f-4aee-4ed2-9355-3cd5222862df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922b063f-4aee-4ed2-9355-3cd5222862df-istio-podinfo\") pod \"922b063f-4aee-4ed2-9355-3cd5222862df\" (UID: \"922b063f-4aee-4ed2-9355-3cd5222862df\") " Oct 31 20:05:24.328553 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.328435 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-credential-socket" pod "922b063f-4aee-4ed2-9355-3cd5222862df" (UID: "922b063f-4aee-4ed2-9355-3cd5222862df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:24.328693 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.328648 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-workload-socket" pod "922b063f-4aee-4ed2-9355-3cd5222862df" (UID: "922b063f-4aee-4ed2-9355-3cd5222862df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:24.328841 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.328814 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/922b063f-4aee-4ed2-9355-3cd5222862df-istiod-ca-cert" pod "922b063f-4aee-4ed2-9355-3cd5222862df" (UID: "922b063f-4aee-4ed2-9355-3cd5222862df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:05:24.329129 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.329109 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:24.329129 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.329129 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:24.329321 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.329140 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/922b063f-4aee-4ed2-9355-3cd5222862df-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:24.329321 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.329155 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:24.329321 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.329168 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:24.331252 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.330895 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922b063f-4aee-4ed2-9355-3cd5222862df-istio-token" pod "922b063f-4aee-4ed2-9355-3cd5222862df" (UID: "922b063f-4aee-4ed2-9355-3cd5222862df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:24.331252 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.331031 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922b063f-4aee-4ed2-9355-3cd5222862df-kube-api-access-97z6l" pod "922b063f-4aee-4ed2-9355-3cd5222862df" (UID: "922b063f-4aee-4ed2-9355-3cd5222862df"). InnerVolumeSpecName "kube-api-access-97z6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:24.331252 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.331074 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-istio-envoy" pod "922b063f-4aee-4ed2-9355-3cd5222862df" (UID: "922b063f-4aee-4ed2-9355-3cd5222862df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:24.331568 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.331553 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/922b063f-4aee-4ed2-9355-3cd5222862df-istio-podinfo" pod "922b063f-4aee-4ed2-9355-3cd5222862df" (UID: "922b063f-4aee-4ed2-9355-3cd5222862df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:05:24.453055 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.450140 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97z6l\" (UniqueName: \"kubernetes.io/projected/922b063f-4aee-4ed2-9355-3cd5222862df-kube-api-access-97z6l\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:24.453055 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.450182 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/922b063f-4aee-4ed2-9355-3cd5222862df-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:24.453055 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.450196 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/922b063f-4aee-4ed2-9355-3cd5222862df-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:24.453055 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.450209 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/922b063f-4aee-4ed2-9355-3cd5222862df-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:24.938988 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.938958 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" Oct 31 20:05:24.939509 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.938988 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc" event={"ID":"922b063f-4aee-4ed2-9355-3cd5222862df","Type":"ContainerDied","Data":"f5eeb8b7349c1f80ec72108d602c24485c3d341a05a718a54733c7867f911ad8"} Oct 31 20:05:24.939509 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.939057 4910 scope.go:122] "RemoveContainer" containerID="ea1885212f37c461eb2ee862b79f1b42373c21f2cf365a087849a56c55ac8550" Oct 31 20:05:24.975580 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.974048 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc"] Oct 31 20:05:24.980214 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:24.980180 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-dmjwc"] Oct 31 20:05:25.150365 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.150339 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b2c5fc-n4b69" Oct 31 20:05:25.209433 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.209360 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d613866-7sb6w" Oct 31 20:05:25.264290 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.264258 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/698675aa-5484-476d-9b7a-e677c312dcc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/698675aa-5484-476d-9b7a-e677c312dcc3-maas-api-openshift-ca\") pod \"698675aa-5484-476d-9b7a-e677c312dcc3\" (UID: \"698675aa-5484-476d-9b7a-e677c312dcc3\") " Oct 31 20:05:25.264485 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.264331 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/698675aa-5484-476d-9b7a-e677c312dcc3-kube-api-access-vq6w9\" (UniqueName: \"kubernetes.io/projected/698675aa-5484-476d-9b7a-e677c312dcc3-kube-api-access-vq6w9\") pod \"698675aa-5484-476d-9b7a-e677c312dcc3\" (UID: \"698675aa-5484-476d-9b7a-e677c312dcc3\") " Oct 31 20:05:25.264485 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.264412 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad7d564e-a00c-4722-9849-dcbd5de16477-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad7d564e-a00c-4722-9849-dcbd5de16477-maas-api-openshift-ca\") pod \"ad7d564e-a00c-4722-9849-dcbd5de16477\" (UID: \"ad7d564e-a00c-4722-9849-dcbd5de16477\") " Oct 31 20:05:25.264625 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.264600 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/698675aa-5484-476d-9b7a-e677c312dcc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/698675aa-5484-476d-9b7a-e677c312dcc3-maas-api-xks-ca\") pod \"698675aa-5484-476d-9b7a-e677c312dcc3\" (UID: \"698675aa-5484-476d-9b7a-e677c312dcc3\") " Oct 31 20:05:25.264697 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.264673 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/698675aa-5484-476d-9b7a-e677c312dcc3-maas-api-openshift-ca" pod "698675aa-5484-476d-9b7a-e677c312dcc3" (UID: "698675aa-5484-476d-9b7a-e677c312dcc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:05:25.264760 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.264719 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad7d564e-a00c-4722-9849-dcbd5de16477-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad7d564e-a00c-4722-9849-dcbd5de16477-maas-api-xks-ca\") pod \"ad7d564e-a00c-4722-9849-dcbd5de16477\" (UID: \"ad7d564e-a00c-4722-9849-dcbd5de16477\") " Oct 31 20:05:25.264816 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.264764 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad7d564e-a00c-4722-9849-dcbd5de16477-kube-api-access-x4422\" (UniqueName: \"kubernetes.io/projected/ad7d564e-a00c-4722-9849-dcbd5de16477-kube-api-access-x4422\") pod \"ad7d564e-a00c-4722-9849-dcbd5de16477\" (UID: \"ad7d564e-a00c-4722-9849-dcbd5de16477\") " Oct 31 20:05:25.265064 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.265040 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad7d564e-a00c-4722-9849-dcbd5de16477-maas-api-openshift-ca" pod "ad7d564e-a00c-4722-9849-dcbd5de16477" (UID: "ad7d564e-a00c-4722-9849-dcbd5de16477"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:05:25.265378 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.265353 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/698675aa-5484-476d-9b7a-e677c312dcc3-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:25.265445 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.265383 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad7d564e-a00c-4722-9849-dcbd5de16477-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:25.267881 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.267839 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad7d564e-a00c-4722-9849-dcbd5de16477-maas-api-xks-ca" pod "ad7d564e-a00c-4722-9849-dcbd5de16477" (UID: "ad7d564e-a00c-4722-9849-dcbd5de16477"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:05:25.268225 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.268077 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad7d564e-a00c-4722-9849-dcbd5de16477-kube-api-access-x4422" pod "ad7d564e-a00c-4722-9849-dcbd5de16477" (UID: "ad7d564e-a00c-4722-9849-dcbd5de16477"). InnerVolumeSpecName "kube-api-access-x4422". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:25.268420 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.268389 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698675aa-5484-476d-9b7a-e677c312dcc3-kube-api-access-vq6w9" pod "698675aa-5484-476d-9b7a-e677c312dcc3" (UID: "698675aa-5484-476d-9b7a-e677c312dcc3"). InnerVolumeSpecName "kube-api-access-vq6w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:25.268635 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.268613 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/698675aa-5484-476d-9b7a-e677c312dcc3-maas-api-xks-ca" pod "698675aa-5484-476d-9b7a-e677c312dcc3" (UID: "698675aa-5484-476d-9b7a-e677c312dcc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:05:25.367239 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.367201 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4422\" (UniqueName: \"kubernetes.io/projected/ad7d564e-a00c-4722-9849-dcbd5de16477-kube-api-access-x4422\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:25.367239 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.367238 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq6w9\" (UniqueName: \"kubernetes.io/projected/698675aa-5484-476d-9b7a-e677c312dcc3-kube-api-access-vq6w9\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:25.367464 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.367252 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/698675aa-5484-476d-9b7a-e677c312dcc3-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:25.367464 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.367267 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad7d564e-a00c-4722-9849-dcbd5de16477-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:25.758967 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.758930 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="922b063f-4aee-4ed2-9355-3cd5222862df" path="/var/lib/kubelet/pods/922b063f-4aee-4ed2-9355-3cd5222862df/volumes" Oct 31 20:05:25.766193 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.766160 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" containerID="cri-o://747142fb9c733d12c2b3edf18268ae5a375df36c7b9927f79472a39c9aed5bde" gracePeriod=45 Oct 31 20:05:25.816039 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.815999 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" containerID="cri-o://672946e1083bcc7da2e7fbec1123b3ec89a7bad2324dd932b88861e0b95d0571" gracePeriod=45 Oct 31 20:05:25.954545 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.954514 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b2c5fc-n4b69" Oct 31 20:05:25.954545 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.954521 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b2c5fc-n4b69" event={"ID":"698675aa-5484-476d-9b7a-e677c312dcc3","Type":"ContainerDied","Data":"2cc9ce129238fee154fbe85173b8959bb69ffa504404757b30524dfaff488c5f"} Oct 31 20:05:25.955028 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.954590 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cc9ce129238fee154fbe85173b8959bb69ffa504404757b30524dfaff488c5f" Oct 31 20:05:25.956626 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.956601 4910 generic.go:356] "Generic (PLEG): container finished" podID="f2588d18-ac6b-403c-a350-11ecf531b468" containerID="672946e1083bcc7da2e7fbec1123b3ec89a7bad2324dd932b88861e0b95d0571" exitCode=0 Oct 31 20:05:25.956747 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.956706 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" event={"ID":"f2588d18-ac6b-403c-a350-11ecf531b468","Type":"ContainerDied","Data":"672946e1083bcc7da2e7fbec1123b3ec89a7bad2324dd932b88861e0b95d0571"} Oct 31 20:05:25.959426 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.958366 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d613866-7sb6w" event={"ID":"ad7d564e-a00c-4722-9849-dcbd5de16477","Type":"ContainerDied","Data":"3bc1dc28fd575a7978f5e00c519802b2140a820aca2fcdea99c4e395619d03ed"} Oct 31 20:05:25.959426 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.958408 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bc1dc28fd575a7978f5e00c519802b2140a820aca2fcdea99c4e395619d03ed" Oct 31 20:05:25.959426 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.958476 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d613866-7sb6w" Oct 31 20:05:25.962615 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.962447 4910 generic.go:356] "Generic (PLEG): container finished" podID="97abf78a-128a-4f23-947f-284d7dfecace" containerID="747142fb9c733d12c2b3edf18268ae5a375df36c7b9927f79472a39c9aed5bde" exitCode=0 Oct 31 20:05:25.962615 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:25.962562 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" event={"ID":"97abf78a-128a-4f23-947f-284d7dfecace","Type":"ContainerDied","Data":"747142fb9c733d12c2b3edf18268ae5a375df36c7b9927f79472a39c9aed5bde"} Oct 31 20:05:26.107857 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.107823 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 20:05:26.162040 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.162004 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 20:05:26.180232 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.180192 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97abf78a-128a-4f23-947f-284d7dfecace-tls-certs\" (UniqueName: \"kubernetes.io/secret/97abf78a-128a-4f23-947f-284d7dfecace-tls-certs\") pod \"97abf78a-128a-4f23-947f-284d7dfecace\" (UID: \"97abf78a-128a-4f23-947f-284d7dfecace\") " Oct 31 20:05:26.180435 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.180251 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-model-cache\") pod \"97abf78a-128a-4f23-947f-284d7dfecace\" (UID: \"97abf78a-128a-4f23-947f-284d7dfecace\") " Oct 31 20:05:26.180435 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.180286 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-home\" (UniqueName: \"kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-home\") pod \"97abf78a-128a-4f23-947f-284d7dfecace\" (UID: \"97abf78a-128a-4f23-947f-284d7dfecace\") " Oct 31 20:05:26.180435 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.180347 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97abf78a-128a-4f23-947f-284d7dfecace-kube-api-access-9jszf\" (UniqueName: \"kubernetes.io/projected/97abf78a-128a-4f23-947f-284d7dfecace-kube-api-access-9jszf\") pod \"97abf78a-128a-4f23-947f-284d7dfecace\" (UID: \"97abf78a-128a-4f23-947f-284d7dfecace\") " Oct 31 20:05:26.180627 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.180494 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-tmp-dir\") pod \"97abf78a-128a-4f23-947f-284d7dfecace\" (UID: \"97abf78a-128a-4f23-947f-284d7dfecace\") " Oct 31 20:05:26.180627 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.180528 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-dshm\") pod \"97abf78a-128a-4f23-947f-284d7dfecace\" (UID: \"97abf78a-128a-4f23-947f-284d7dfecace\") " Oct 31 20:05:26.180627 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.180611 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-home" pod "97abf78a-128a-4f23-947f-284d7dfecace" (UID: "97abf78a-128a-4f23-947f-284d7dfecace"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:26.180796 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.180744 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-tmp-dir" pod "97abf78a-128a-4f23-947f-284d7dfecace" (UID: "97abf78a-128a-4f23-947f-284d7dfecace"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:26.181174 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.181149 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-model-cache" pod "97abf78a-128a-4f23-947f-284d7dfecace" (UID: "97abf78a-128a-4f23-947f-284d7dfecace"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:26.181290 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.181209 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-home\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:26.181290 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.181229 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-tmp-dir\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:26.183340 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.183310 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97abf78a-128a-4f23-947f-284d7dfecace-tls-certs" pod "97abf78a-128a-4f23-947f-284d7dfecace" (UID: "97abf78a-128a-4f23-947f-284d7dfecace"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:05:26.183590 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.183569 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-dshm" pod "97abf78a-128a-4f23-947f-284d7dfecace" (UID: "97abf78a-128a-4f23-947f-284d7dfecace"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:26.191649 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.191290 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97abf78a-128a-4f23-947f-284d7dfecace-kube-api-access-9jszf" pod "97abf78a-128a-4f23-947f-284d7dfecace" (UID: "97abf78a-128a-4f23-947f-284d7dfecace"). InnerVolumeSpecName "kube-api-access-9jszf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:26.281989 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.281889 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-dshm\") pod \"f2588d18-ac6b-403c-a350-11ecf531b468\" (UID: \"f2588d18-ac6b-403c-a350-11ecf531b468\") " Oct 31 20:05:26.282182 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.282047 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-tmp-dir\") pod \"f2588d18-ac6b-403c-a350-11ecf531b468\" (UID: \"f2588d18-ac6b-403c-a350-11ecf531b468\") " Oct 31 20:05:26.282182 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.282095 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-home\" (UniqueName: \"kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-home\") pod \"f2588d18-ac6b-403c-a350-11ecf531b468\" (UID: \"f2588d18-ac6b-403c-a350-11ecf531b468\") " Oct 31 20:05:26.282182 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.282142 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2588d18-ac6b-403c-a350-11ecf531b468-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2588d18-ac6b-403c-a350-11ecf531b468-tls-certs\") pod \"f2588d18-ac6b-403c-a350-11ecf531b468\" (UID: \"f2588d18-ac6b-403c-a350-11ecf531b468\") " Oct 31 20:05:26.282182 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.282169 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2588d18-ac6b-403c-a350-11ecf531b468-kube-api-access-wkck6\" (UniqueName: \"kubernetes.io/projected/f2588d18-ac6b-403c-a350-11ecf531b468-kube-api-access-wkck6\") pod \"f2588d18-ac6b-403c-a350-11ecf531b468\" (UID: \"f2588d18-ac6b-403c-a350-11ecf531b468\") " Oct 31 20:05:26.282410 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.282210 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-model-cache\") pod \"f2588d18-ac6b-403c-a350-11ecf531b468\" (UID: \"f2588d18-ac6b-403c-a350-11ecf531b468\") " Oct 31 20:05:26.282479 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.282420 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-home" pod "f2588d18-ac6b-403c-a350-11ecf531b468" (UID: "f2588d18-ac6b-403c-a350-11ecf531b468"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:26.282543 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.282519 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-model-cache" pod "f2588d18-ac6b-403c-a350-11ecf531b468" (UID: "f2588d18-ac6b-403c-a350-11ecf531b468"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:26.282904 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.282877 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-dshm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:26.283062 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.282947 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-home\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:26.283062 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.282964 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-model-cache\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:26.283062 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.282982 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97abf78a-128a-4f23-947f-284d7dfecace-tls-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:26.283062 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.282996 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97abf78a-128a-4f23-947f-284d7dfecace-model-cache\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:26.283062 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.283011 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jszf\" (UniqueName: \"kubernetes.io/projected/97abf78a-128a-4f23-947f-284d7dfecace-kube-api-access-9jszf\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:26.283062 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.282874 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-tmp-dir" pod "f2588d18-ac6b-403c-a350-11ecf531b468" (UID: "f2588d18-ac6b-403c-a350-11ecf531b468"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:26.284467 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.284440 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2588d18-ac6b-403c-a350-11ecf531b468-tls-certs" pod "f2588d18-ac6b-403c-a350-11ecf531b468" (UID: "f2588d18-ac6b-403c-a350-11ecf531b468"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:05:26.284566 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.284485 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-dshm" pod "f2588d18-ac6b-403c-a350-11ecf531b468" (UID: "f2588d18-ac6b-403c-a350-11ecf531b468"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:26.284566 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.284531 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2588d18-ac6b-403c-a350-11ecf531b468-kube-api-access-wkck6" pod "f2588d18-ac6b-403c-a350-11ecf531b468" (UID: "f2588d18-ac6b-403c-a350-11ecf531b468"). InnerVolumeSpecName "kube-api-access-wkck6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:26.384473 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.384433 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-tmp-dir\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:26.384473 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.384467 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2588d18-ac6b-403c-a350-11ecf531b468-tls-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:26.384473 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.384477 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkck6\" (UniqueName: \"kubernetes.io/projected/f2588d18-ac6b-403c-a350-11ecf531b468-kube-api-access-wkck6\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:26.384770 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.384486 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2588d18-ac6b-403c-a350-11ecf531b468-dshm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:26.974591 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.974564 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" Oct 31 20:05:26.975038 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.974592 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv" event={"ID":"f2588d18-ac6b-403c-a350-11ecf531b468","Type":"ContainerDied","Data":"8dd3210d637854ace4003879f1edf6cb2c24730adaf68c175add37586863e3bd"} Oct 31 20:05:26.975038 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.974647 4910 scope.go:122] "RemoveContainer" containerID="672946e1083bcc7da2e7fbec1123b3ec89a7bad2324dd932b88861e0b95d0571" Oct 31 20:05:26.976367 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.976343 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" event={"ID":"97abf78a-128a-4f23-947f-284d7dfecace","Type":"ContainerDied","Data":"ff47317338a28fd8598fba6b20fac757cd28bbf375ee09b8d3ab41a04a090ee2"} Oct 31 20:05:26.976562 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.976376 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt" Oct 31 20:05:26.987379 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:26.987359 4910 scope.go:122] "RemoveContainer" containerID="747142fb9c733d12c2b3edf18268ae5a375df36c7b9927f79472a39c9aed5bde" Oct 31 20:05:27.009267 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:27.009241 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv"] Oct 31 20:05:27.014274 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:27.014243 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-996c32/simulator-w4-996c32-kserve-d747cf979-rf6sv"] Oct 31 20:05:27.031971 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:27.031937 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt"] Oct 31 20:05:27.035250 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:27.035223 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-996c32/distinct-2-w4-996c32-kserve-6d8dc6498-wgxjt"] Oct 31 20:05:27.762932 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:27.762315 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97abf78a-128a-4f23-947f-284d7dfecace" path="/var/lib/kubelet/pods/97abf78a-128a-4f23-947f-284d7dfecace/volumes" Oct 31 20:05:27.763140 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:27.763048 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" path="/var/lib/kubelet/pods/f2588d18-ac6b-403c-a350-11ecf531b468/volumes" Oct 31 20:05:29.465836 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:29.465792 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w"] Oct 31 20:05:29.472963 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:29.471130 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-e1d060-2x69w"] Oct 31 20:05:29.792238 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:29.792141 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eee6f79c-82ab-4c26-abb9-038ad03a9dc0" path="/var/lib/kubelet/pods/eee6f79c-82ab-4c26-abb9-038ad03a9dc0/volumes" Oct 31 20:05:30.980441 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:30.971499 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j"] Oct 31 20:05:30.980441 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:30.972370 4910 cpu_manager.go:414] "containerMap: removing container" podUID="698675aa-5484-476d-9b7a-e677c312dcc3" containerName="revoke-keys" Oct 31 20:05:30.980441 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:30.972393 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="698675aa-5484-476d-9b7a-e677c312dcc3" containerName="revoke-keys" Oct 31 20:05:30.980441 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:30.972422 4910 cpu_manager.go:414] "containerMap: removing container" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" Oct 31 20:05:30.980441 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:30.972429 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" Oct 31 20:05:30.980441 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:30.972447 4910 cpu_manager.go:414] "containerMap: removing container" podUID="ce05955e-1925-4469-978f-29f94fc3d6f0" containerName="istio-proxy" Oct 31 20:05:30.980441 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:30.972455 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce05955e-1925-4469-978f-29f94fc3d6f0" containerName="istio-proxy" Oct 31 20:05:30.980441 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:30.972491 4910 cpu_manager.go:414] "containerMap: removing container" podUID="59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13" containerName="istio-proxy" Oct 31 20:05:30.980441 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:30.972498 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13" containerName="istio-proxy" Oct 31 20:05:30.980441 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:30.972515 4910 cpu_manager.go:414] "containerMap: removing container" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" Oct 31 20:05:30.980441 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:30.972523 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" Oct 31 20:05:30.980441 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:30.972539 4910 cpu_manager.go:414] "containerMap: removing container" podUID="922b063f-4aee-4ed2-9355-3cd5222862df" containerName="istio-proxy" Oct 31 20:05:30.980441 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:30.972546 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="922b063f-4aee-4ed2-9355-3cd5222862df" containerName="istio-proxy" Oct 31 20:05:30.980441 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:30.972557 4910 cpu_manager.go:414] "containerMap: removing container" podUID="ad7d564e-a00c-4722-9849-dcbd5de16477" containerName="revoke-keys" Oct 31 20:05:30.980441 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:30.972564 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad7d564e-a00c-4722-9849-dcbd5de16477" containerName="revoke-keys" Oct 31 20:05:30.980441 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:30.972702 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce05955e-1925-4469-978f-29f94fc3d6f0" containerName="istio-proxy" Oct 31 20:05:30.980441 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:30.972722 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="59ffdaa6-2c15-4417-a54e-3dcaa4fa7f13" containerName="istio-proxy" Oct 31 20:05:30.980441 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:30.972733 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="97abf78a-128a-4f23-947f-284d7dfecace" containerName="main" Oct 31 20:05:30.980441 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:30.972743 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="922b063f-4aee-4ed2-9355-3cd5222862df" containerName="istio-proxy" Oct 31 20:05:30.980441 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:30.972754 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad7d564e-a00c-4722-9849-dcbd5de16477" containerName="revoke-keys" Oct 31 20:05:30.980441 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:30.972763 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="922b063f-4aee-4ed2-9355-3cd5222862df" containerName="istio-proxy" Oct 31 20:05:30.980441 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:30.972775 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="698675aa-5484-476d-9b7a-e677c312dcc3" containerName="revoke-keys" Oct 31 20:05:30.980441 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:30.972788 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2588d18-ac6b-403c-a350-11ecf531b468" containerName="main" Oct 31 20:05:30.982085 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:30.981774 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.003790 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.000700 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f28d48-openshift-default-dockercfg-mg2rh\"" Oct 31 20:05:31.073758 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.073725 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j"] Oct 31 20:05:31.116243 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.116153 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-podinfo\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.116399 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.116248 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9136da5b-c568-412a-b262-eb85bb4cf6b6-istiod-ca-cert\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.116399 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.116308 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-workload-certs\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.116502 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.116444 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-data\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.116537 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.116499 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-credential-socket\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.116699 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.116678 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-workload-socket\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.116742 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.116718 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-token\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.116848 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.116830 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-envoy\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.117017 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.116997 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpw9x\" (UniqueName: \"kubernetes.io/projected/9136da5b-c568-412a-b262-eb85bb4cf6b6-kube-api-access-mpw9x\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.218816 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.218776 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-envoy\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.219048 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.218905 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpw9x\" (UniqueName: \"kubernetes.io/projected/9136da5b-c568-412a-b262-eb85bb4cf6b6-kube-api-access-mpw9x\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.219048 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.218994 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-podinfo\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.219198 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.219048 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9136da5b-c568-412a-b262-eb85bb4cf6b6-istiod-ca-cert\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.219198 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.219074 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-workload-certs\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.219198 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.219145 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-data\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.219362 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.219225 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-credential-socket\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.219362 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.219338 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-workload-socket\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.219469 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.219367 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-token\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.219887 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.219536 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-workload-certs\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.219887 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.219628 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-data\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.219887 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.219846 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-workload-socket\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.220131 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.220041 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9136da5b-c568-412a-b262-eb85bb4cf6b6-istiod-ca-cert\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.220477 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.220447 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-credential-socket\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.221594 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.221574 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-envoy\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.221778 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.221756 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-podinfo\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.234306 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.234236 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpw9x\" (UniqueName: \"kubernetes.io/projected/9136da5b-c568-412a-b262-eb85bb4cf6b6-kube-api-access-mpw9x\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.235164 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.235144 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-token\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.311174 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.311139 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:31.627095 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.627052 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j"] Oct 31 20:05:31.629734 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:05:31.629696 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9136da5b_c568_412a_b262_eb85bb4cf6b6.slice/crio-865fe27c74bf59a556c58a53446fc8f27dc6c052d6fdbfe809e1a38def52fb47 WatchSource:0}: Error finding container 865fe27c74bf59a556c58a53446fc8f27dc6c052d6fdbfe809e1a38def52fb47: Status 404 returned error can't find the container with id 865fe27c74bf59a556c58a53446fc8f27dc6c052d6fdbfe809e1a38def52fb47 Oct 31 20:05:31.633199 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.633147 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:05:31.633313 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.633280 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:05:31.633372 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.633322 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:05:31.865194 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.865154 4910 scope.go:122] "RemoveContainer" containerID="8150b25104a272dcc02b0c705ba81a15558e80934d66731684cb5b3a303cf963" Oct 31 20:05:31.876776 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:31.876754 4910 scope.go:122] "RemoveContainer" containerID="30f8a31b226e258d422416032273e363eef75e9a5b2680dffb5e1a617de93f59" Oct 31 20:05:32.084581 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:32.084548 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr"] Oct 31 20:05:32.085153 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:32.085139 4910 cpu_manager.go:414] "containerMap: removing container" podUID="922b063f-4aee-4ed2-9355-3cd5222862df" containerName="istio-proxy" Oct 31 20:05:32.085211 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:32.085154 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="922b063f-4aee-4ed2-9355-3cd5222862df" containerName="istio-proxy" Oct 31 20:05:32.090536 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:32.090515 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr" Oct 31 20:05:32.107012 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:32.106984 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr"] Oct 31 20:05:32.112329 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:32.112299 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:05:32.112647 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:32.112627 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 20:05:32.122498 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:32.122464 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" event={"ID":"9136da5b-c568-412a-b262-eb85bb4cf6b6","Type":"ContainerStarted","Data":"c85ca61f3d3007aa3e13f0f199caf5705557081a37e6fb41cb13462f456a7f66"} Oct 31 20:05:32.122649 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:32.122525 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" event={"ID":"9136da5b-c568-412a-b262-eb85bb4cf6b6","Type":"ContainerStarted","Data":"865fe27c74bf59a556c58a53446fc8f27dc6c052d6fdbfe809e1a38def52fb47"} Oct 31 20:05:32.165214 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:32.164933 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" podStartSLOduration=2.164898356 podStartE2EDuration="2.164898356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:32.156721386 +0000 UTC m=+2766.519541759" watchObservedRunningTime="2026-10-31 20:05:32.164898356 +0000 UTC m=+2766.527718729" Oct 31 20:05:32.234410 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:32.234373 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr\" (UID: \"aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr" Oct 31 20:05:32.234837 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:32.234807 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr\" (UID: \"aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr" Oct 31 20:05:32.235192 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:32.235133 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzkzh\" (UniqueName: \"kubernetes.io/projected/aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79-kube-api-access-pzkzh\") pod \"maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr\" (UID: \"aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr" Oct 31 20:05:32.311845 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:32.311806 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:32.313733 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:32.313701 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 31 20:05:32.313871 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:32.313758 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" podUID="9136da5b-c568-412a-b262-eb85bb4cf6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 31 20:05:32.336530 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:32.336492 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr\" (UID: \"aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr" Oct 31 20:05:32.336670 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:32.336602 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzkzh\" (UniqueName: \"kubernetes.io/projected/aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79-kube-api-access-pzkzh\") pod \"maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr\" (UID: \"aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr" Oct 31 20:05:32.336787 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:32.336772 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr\" (UID: \"aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr" Oct 31 20:05:32.337229 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:32.337199 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr\" (UID: \"aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr" Oct 31 20:05:32.339288 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:32.339255 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr\" (UID: \"aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr" Oct 31 20:05:32.353827 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:32.353798 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzkzh\" (UniqueName: \"kubernetes.io/projected/aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79-kube-api-access-pzkzh\") pod \"maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr\" (UID: \"aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr" Oct 31 20:05:32.406968 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:32.406895 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr" Oct 31 20:05:32.709109 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:05:32.709079 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa6e068f_8ca6_4ce8_91ce_e5d324c5bf79.slice/crio-8ca5ed92ecb9dfd04fd13f8d3820257c7db8f7cb643bb0004be6b6dc79c2a079 WatchSource:0}: Error finding container 8ca5ed92ecb9dfd04fd13f8d3820257c7db8f7cb643bb0004be6b6dc79c2a079: Status 404 returned error can't find the container with id 8ca5ed92ecb9dfd04fd13f8d3820257c7db8f7cb643bb0004be6b6dc79c2a079 Oct 31 20:05:32.709211 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:32.709109 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr"] Oct 31 20:05:33.160157 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:33.160122 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr" event={"ID":"aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79","Type":"ContainerStarted","Data":"8ca5ed92ecb9dfd04fd13f8d3820257c7db8f7cb643bb0004be6b6dc79c2a079"} Oct 31 20:05:33.313049 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:33.313011 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 31 20:05:33.313230 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:33.313066 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" podUID="9136da5b-c568-412a-b262-eb85bb4cf6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 31 20:05:34.170851 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:34.170819 4910 generic.go:356] "Generic (PLEG): container finished" podID="aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79" containerID="0bb99fa894b6ae8dfd8f40536e6730c045922777b68d607a2fe62fbb8977c6a7" exitCode=0 Oct 31 20:05:34.171246 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:34.170906 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr" event={"ID":"aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79","Type":"ContainerDied","Data":"0bb99fa894b6ae8dfd8f40536e6730c045922777b68d607a2fe62fbb8977c6a7"} Oct 31 20:05:34.312287 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:34.312259 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 31 20:05:34.312463 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:34.312325 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" podUID="9136da5b-c568-412a-b262-eb85bb4cf6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 31 20:05:35.313884 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:35.313830 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 31 20:05:35.314537 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:35.313921 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" podUID="9136da5b-c568-412a-b262-eb85bb4cf6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 31 20:05:35.410075 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:35.410049 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr" Oct 31 20:05:35.557180 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:35.540406 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79-maas-api-xks-ca\") pod \"aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79\" (UID: \"aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79\") " Oct 31 20:05:35.557180 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:35.540471 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79-maas-api-openshift-ca\") pod \"aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79\" (UID: \"aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79\") " Oct 31 20:05:35.557180 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:35.540539 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79-kube-api-access-pzkzh\" (UniqueName: \"kubernetes.io/projected/aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79-kube-api-access-pzkzh\") pod \"aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79\" (UID: \"aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79\") " Oct 31 20:05:35.558460 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:35.557959 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79-maas-api-xks-ca" pod "aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79" (UID: "aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:05:35.558460 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:35.558099 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79-kube-api-access-pzkzh" pod "aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79" (UID: "aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79"). InnerVolumeSpecName "kube-api-access-pzkzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:35.558460 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:35.558260 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79-maas-api-openshift-ca" pod "aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79" (UID: "aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:05:35.643554 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:35.643451 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:35.643554 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:35.643491 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:35.643554 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:35.643510 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzkzh\" (UniqueName: \"kubernetes.io/projected/aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79-kube-api-access-pzkzh\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:36.194873 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:36.194002 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr" Oct 31 20:05:36.194873 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:36.194083 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr" event={"ID":"aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79","Type":"ContainerDied","Data":"8ca5ed92ecb9dfd04fd13f8d3820257c7db8f7cb643bb0004be6b6dc79c2a079"} Oct 31 20:05:36.194873 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:36.194145 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ca5ed92ecb9dfd04fd13f8d3820257c7db8f7cb643bb0004be6b6dc79c2a079" Oct 31 20:05:36.312875 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:36.312541 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 31 20:05:36.312875 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:36.312595 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" podUID="9136da5b-c568-412a-b262-eb85bb4cf6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 31 20:05:37.312504 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:37.312463 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Oct 31 20:05:37.312952 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:37.312527 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" podUID="9136da5b-c568-412a-b262-eb85bb4cf6b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Oct 31 20:05:38.525943 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:38.525896 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:38.528095 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:38.527957 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:38.532200 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:38.532172 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:05:39.151597 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:39.151555 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd"] Oct 31 20:05:39.152142 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:39.152102 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" podUID="e898f0b3-1c73-44d5-bc56-1d2c5673d516" containerName="istio-proxy" containerID="cri-o://138e83d447d96ef6fee84b68ec56c175b342faa1e7eed49813a9c169a03e1cf2" gracePeriod=30 Oct 31 20:05:40.363282 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:40.363244 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5"] Oct 31 20:05:40.374270 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:40.374241 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-996c32-grcc5"] Oct 31 20:05:41.062937 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:41.061331 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr"] Oct 31 20:05:41.062937 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:41.061557 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" podUID="ceb11f75-3d05-4e9c-8db7-caa224621cad" containerName="istio-proxy" containerID="cri-o://fedbdb204387b3305e30411da5ecfa09c5282cfb45d84d787e225acd28eac397" gracePeriod=30 Oct 31 20:05:41.107008 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:41.106718 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2"] Oct 31 20:05:41.109005 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:41.107432 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" podUID="949b61a5-f07e-47c1-b360-a3bfde012f42" containerName="istio-proxy" containerID="cri-o://771ee87af4ec83f3266431bf0447e56a1d0d18820ff12724e8d48d0c100dabac" gracePeriod=30 Oct 31 20:05:41.112797 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:41.112731 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm"] Oct 31 20:05:41.113181 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:41.113035 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" podUID="f41d2753-d5fe-4a08-a4e4-180c1c3c5972" containerName="istio-proxy" containerID="cri-o://34872d8e140d477d4e1acaff2892d5365efd05027a5e8755d308114fc0503dae" gracePeriod=30 Oct 31 20:05:41.645973 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:41.645904 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:05:41.646433 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:41.645990 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" podUID="ceb11f75-3d05-4e9c-8db7-caa224621cad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:05:41.785175 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:41.785134 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1234fff-2cf6-40b7-ab83-bd37f5a75369" path="/var/lib/kubelet/pods/a1234fff-2cf6-40b7-ab83-bd37f5a75369/volumes" Oct 31 20:05:41.785662 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:41.785642 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn"] Oct 31 20:05:41.786496 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:41.786476 4910 cpu_manager.go:414] "containerMap: removing container" podUID="aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79" containerName="revoke-keys" Oct 31 20:05:41.786546 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:41.786499 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79" containerName="revoke-keys" Oct 31 20:05:41.786688 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:41.786674 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79" containerName="revoke-keys" Oct 31 20:05:41.792598 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:41.792376 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn" Oct 31 20:05:41.798238 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:41.798213 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn"] Oct 31 20:05:41.805281 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:41.804448 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:05:41.807453 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:41.807392 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 20:05:41.915375 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:41.915286 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a673180d-8ebc-474d-a6d1-130ea7603aed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn\" (UID: \"a673180d-8ebc-474d-a6d1-130ea7603aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn" Oct 31 20:05:41.915687 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:41.915655 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a673180d-8ebc-474d-a6d1-130ea7603aed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn\" (UID: \"a673180d-8ebc-474d-a6d1-130ea7603aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn" Oct 31 20:05:41.915749 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:41.915706 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q5mk\" (UniqueName: \"kubernetes.io/projected/a673180d-8ebc-474d-a6d1-130ea7603aed-kube-api-access-4q5mk\") pod \"maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn\" (UID: \"a673180d-8ebc-474d-a6d1-130ea7603aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn" Oct 31 20:05:42.016708 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:42.016669 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a673180d-8ebc-474d-a6d1-130ea7603aed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn\" (UID: \"a673180d-8ebc-474d-a6d1-130ea7603aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn" Oct 31 20:05:42.016708 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:42.016708 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q5mk\" (UniqueName: \"kubernetes.io/projected/a673180d-8ebc-474d-a6d1-130ea7603aed-kube-api-access-4q5mk\") pod \"maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn\" (UID: \"a673180d-8ebc-474d-a6d1-130ea7603aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn" Oct 31 20:05:42.017333 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:42.016847 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a673180d-8ebc-474d-a6d1-130ea7603aed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn\" (UID: \"a673180d-8ebc-474d-a6d1-130ea7603aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn" Oct 31 20:05:42.017401 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:42.017382 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a673180d-8ebc-474d-a6d1-130ea7603aed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn\" (UID: \"a673180d-8ebc-474d-a6d1-130ea7603aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn" Oct 31 20:05:42.019234 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:42.019210 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a673180d-8ebc-474d-a6d1-130ea7603aed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn\" (UID: \"a673180d-8ebc-474d-a6d1-130ea7603aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn" Oct 31 20:05:42.032170 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:42.032137 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q5mk\" (UniqueName: \"kubernetes.io/projected/a673180d-8ebc-474d-a6d1-130ea7603aed-kube-api-access-4q5mk\") pod \"maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn\" (UID: \"a673180d-8ebc-474d-a6d1-130ea7603aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn" Oct 31 20:05:42.110546 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:42.110512 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn" Oct 31 20:05:42.397711 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:42.397677 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn"] Oct 31 20:05:42.397847 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:05:42.397814 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda673180d_8ebc_474d_a6d1_130ea7603aed.slice/crio-e89e0ccf109a15076d698512112188c3e3db905b4fe724acadb78a581ced39fc WatchSource:0}: Error finding container e89e0ccf109a15076d698512112188c3e3db905b4fe724acadb78a581ced39fc: Status 404 returned error can't find the container with id e89e0ccf109a15076d698512112188c3e3db905b4fe724acadb78a581ced39fc Oct 31 20:05:42.714155 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:05:42.714067 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:05:43.308950 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:43.308895 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn" event={"ID":"a673180d-8ebc-474d-a6d1-130ea7603aed","Type":"ContainerStarted","Data":"0d9dbffa13c8f19a72c6700025afc010569036dcd2de411335b5f2686a75619f"} Oct 31 20:05:43.309172 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:43.308973 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn" event={"ID":"a673180d-8ebc-474d-a6d1-130ea7603aed","Type":"ContainerStarted","Data":"e89e0ccf109a15076d698512112188c3e3db905b4fe724acadb78a581ced39fc"} Oct 31 20:05:43.333676 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:43.333573 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn" podStartSLOduration=2.333557971 podStartE2EDuration="2.333557971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:43.332436083 +0000 UTC m=+2777.695256458" watchObservedRunningTime="2026-10-31 20:05:43.333557971 +0000 UTC m=+2777.696378400" Oct 31 20:05:43.499106 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:43.499068 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:05:43.499278 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:43.499121 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" podUID="f41d2753-d5fe-4a08-a4e4-180c1c3c5972" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:05:43.703291 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:43.703197 4910 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:05:43.703291 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:43.703272 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" podUID="e898f0b3-1c73-44d5-bc56-1d2c5673d516" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:05:44.327506 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.327341 4910 generic.go:356] "Generic (PLEG): container finished" podID="e898f0b3-1c73-44d5-bc56-1d2c5673d516" containerID="138e83d447d96ef6fee84b68ec56c175b342faa1e7eed49813a9c169a03e1cf2" exitCode=0 Oct 31 20:05:44.327506 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.327433 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" event={"ID":"e898f0b3-1c73-44d5-bc56-1d2c5673d516","Type":"ContainerDied","Data":"138e83d447d96ef6fee84b68ec56c175b342faa1e7eed49813a9c169a03e1cf2"} Oct 31 20:05:44.329446 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.329418 4910 generic.go:356] "Generic (PLEG): container finished" podID="a673180d-8ebc-474d-a6d1-130ea7603aed" containerID="0d9dbffa13c8f19a72c6700025afc010569036dcd2de411335b5f2686a75619f" exitCode=0 Oct 31 20:05:44.329611 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.329496 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn" event={"ID":"a673180d-8ebc-474d-a6d1-130ea7603aed","Type":"ContainerDied","Data":"0d9dbffa13c8f19a72c6700025afc010569036dcd2de411335b5f2686a75619f"} Oct 31 20:05:44.608661 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.608632 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 20:05:44.625147 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.625116 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8"] Oct 31 20:05:44.653752 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.653722 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-envoy\") pod \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " Oct 31 20:05:44.653985 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.653798 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-token\" (UniqueName: \"kubernetes.io/projected/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-token\") pod \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " Oct 31 20:05:44.653985 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.653827 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-workload-certs\") pod \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " Oct 31 20:05:44.653985 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.653902 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istiod-ca-cert\") pod \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " Oct 31 20:05:44.654163 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.653992 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-workload-socket\") pod \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " Oct 31 20:05:44.654163 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.654022 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-data\") pod \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " Oct 31 20:05:44.654163 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.654060 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-credential-socket\") pod \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " Oct 31 20:05:44.654729 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.654687 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-workload-certs" pod "e898f0b3-1c73-44d5-bc56-1d2c5673d516" (UID: "e898f0b3-1c73-44d5-bc56-1d2c5673d516"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:44.654879 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.654794 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-credential-socket" pod "e898f0b3-1c73-44d5-bc56-1d2c5673d516" (UID: "e898f0b3-1c73-44d5-bc56-1d2c5673d516"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:44.655569 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.655541 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-data" pod "e898f0b3-1c73-44d5-bc56-1d2c5673d516" (UID: "e898f0b3-1c73-44d5-bc56-1d2c5673d516"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:44.655771 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.655750 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istiod-ca-cert" pod "e898f0b3-1c73-44d5-bc56-1d2c5673d516" (UID: "e898f0b3-1c73-44d5-bc56-1d2c5673d516"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:05:44.656461 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.656269 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-workload-socket" pod "e898f0b3-1c73-44d5-bc56-1d2c5673d516" (UID: "e898f0b3-1c73-44d5-bc56-1d2c5673d516"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:44.669848 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.669808 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7"] Oct 31 20:05:44.672505 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.672456 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-envoy" pod "e898f0b3-1c73-44d5-bc56-1d2c5673d516" (UID: "e898f0b3-1c73-44d5-bc56-1d2c5673d516"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:44.676431 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.676386 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-token" pod "e898f0b3-1c73-44d5-bc56-1d2c5673d516" (UID: "e898f0b3-1c73-44d5-bc56-1d2c5673d516"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:44.755708 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.755661 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-podinfo\") pod \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " Oct 31 20:05:44.755927 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.755718 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e898f0b3-1c73-44d5-bc56-1d2c5673d516-kube-api-access-q9zd7\" (UniqueName: \"kubernetes.io/projected/e898f0b3-1c73-44d5-bc56-1d2c5673d516-kube-api-access-q9zd7\") pod \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\" (UID: \"e898f0b3-1c73-44d5-bc56-1d2c5673d516\") " Oct 31 20:05:44.756562 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.756533 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:44.756684 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.756564 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:44.756684 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.756580 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:44.756684 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.756595 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:44.756684 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.756610 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:44.756684 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.756624 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e898f0b3-1c73-44d5-bc56-1d2c5673d516-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:44.756684 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.756638 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:44.757848 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.757814 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-podinfo" pod "e898f0b3-1c73-44d5-bc56-1d2c5673d516" (UID: "e898f0b3-1c73-44d5-bc56-1d2c5673d516"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:05:44.757975 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.757900 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e898f0b3-1c73-44d5-bc56-1d2c5673d516-kube-api-access-q9zd7" pod "e898f0b3-1c73-44d5-bc56-1d2c5673d516" (UID: "e898f0b3-1c73-44d5-bc56-1d2c5673d516"). InnerVolumeSpecName "kube-api-access-q9zd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:44.857690 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.857598 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e898f0b3-1c73-44d5-bc56-1d2c5673d516-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:44.857690 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:44.857634 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9zd7\" (UniqueName: \"kubernetes.io/projected/e898f0b3-1c73-44d5-bc56-1d2c5673d516-kube-api-access-q9zd7\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:45.340284 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:45.340255 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" Oct 31 20:05:45.340765 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:45.340285 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd" event={"ID":"e898f0b3-1c73-44d5-bc56-1d2c5673d516","Type":"ContainerDied","Data":"11f6280707241f19fff2092eee0ae9b557560cdabc5869bfabea8103be45c19a"} Oct 31 20:05:45.340765 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:45.340356 4910 scope.go:122] "RemoveContainer" containerID="138e83d447d96ef6fee84b68ec56c175b342faa1e7eed49813a9c169a03e1cf2" Oct 31 20:05:45.374561 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:45.374497 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd"] Oct 31 20:05:45.379624 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:45.379591 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-cj6sd"] Oct 31 20:05:45.508386 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:45.508362 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn" Oct 31 20:05:45.566244 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:45.566204 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a673180d-8ebc-474d-a6d1-130ea7603aed-kube-api-access-4q5mk\" (UniqueName: \"kubernetes.io/projected/a673180d-8ebc-474d-a6d1-130ea7603aed-kube-api-access-4q5mk\") pod \"a673180d-8ebc-474d-a6d1-130ea7603aed\" (UID: \"a673180d-8ebc-474d-a6d1-130ea7603aed\") " Oct 31 20:05:45.566510 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:45.566426 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a673180d-8ebc-474d-a6d1-130ea7603aed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a673180d-8ebc-474d-a6d1-130ea7603aed-maas-api-xks-ca\") pod \"a673180d-8ebc-474d-a6d1-130ea7603aed\" (UID: \"a673180d-8ebc-474d-a6d1-130ea7603aed\") " Oct 31 20:05:45.566588 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:45.566554 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a673180d-8ebc-474d-a6d1-130ea7603aed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a673180d-8ebc-474d-a6d1-130ea7603aed-maas-api-openshift-ca\") pod \"a673180d-8ebc-474d-a6d1-130ea7603aed\" (UID: \"a673180d-8ebc-474d-a6d1-130ea7603aed\") " Oct 31 20:05:45.567380 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:45.567347 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a673180d-8ebc-474d-a6d1-130ea7603aed-maas-api-openshift-ca" pod "a673180d-8ebc-474d-a6d1-130ea7603aed" (UID: "a673180d-8ebc-474d-a6d1-130ea7603aed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:05:45.569127 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:45.569101 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a673180d-8ebc-474d-a6d1-130ea7603aed-maas-api-xks-ca" pod "a673180d-8ebc-474d-a6d1-130ea7603aed" (UID: "a673180d-8ebc-474d-a6d1-130ea7603aed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:05:45.569365 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:45.569338 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a673180d-8ebc-474d-a6d1-130ea7603aed-kube-api-access-4q5mk" pod "a673180d-8ebc-474d-a6d1-130ea7603aed" (UID: "a673180d-8ebc-474d-a6d1-130ea7603aed"). InnerVolumeSpecName "kube-api-access-4q5mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:45.668750 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:45.668714 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a673180d-8ebc-474d-a6d1-130ea7603aed-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:45.668750 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:45.668743 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q5mk\" (UniqueName: \"kubernetes.io/projected/a673180d-8ebc-474d-a6d1-130ea7603aed-kube-api-access-4q5mk\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:45.668750 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:45.668755 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a673180d-8ebc-474d-a6d1-130ea7603aed-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:45.757539 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:45.757456 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e898f0b3-1c73-44d5-bc56-1d2c5673d516" path="/var/lib/kubelet/pods/e898f0b3-1c73-44d5-bc56-1d2c5673d516/volumes" Oct 31 20:05:46.357626 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.357461 4910 generic.go:356] "Generic (PLEG): container finished" podID="ceb11f75-3d05-4e9c-8db7-caa224621cad" containerID="fedbdb204387b3305e30411da5ecfa09c5282cfb45d84d787e225acd28eac397" exitCode=0 Oct 31 20:05:46.359256 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.359130 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" event={"ID":"ceb11f75-3d05-4e9c-8db7-caa224621cad","Type":"ContainerDied","Data":"fedbdb204387b3305e30411da5ecfa09c5282cfb45d84d787e225acd28eac397"} Oct 31 20:05:46.363081 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.362845 4910 generic.go:356] "Generic (PLEG): container finished" podID="949b61a5-f07e-47c1-b360-a3bfde012f42" containerID="771ee87af4ec83f3266431bf0447e56a1d0d18820ff12724e8d48d0c100dabac" exitCode=0 Oct 31 20:05:46.363081 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.362951 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" event={"ID":"949b61a5-f07e-47c1-b360-a3bfde012f42","Type":"ContainerDied","Data":"771ee87af4ec83f3266431bf0447e56a1d0d18820ff12724e8d48d0c100dabac"} Oct 31 20:05:46.363081 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.363001 4910 scope.go:122] "RemoveContainer" containerID="77cc12100e7999d67458362233f0ea318c168ef7b636d85a2cdef5eafe159689" Oct 31 20:05:46.370013 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.369988 4910 generic.go:356] "Generic (PLEG): container finished" podID="f41d2753-d5fe-4a08-a4e4-180c1c3c5972" containerID="34872d8e140d477d4e1acaff2892d5365efd05027a5e8755d308114fc0503dae" exitCode=0 Oct 31 20:05:46.370327 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.370307 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" event={"ID":"f41d2753-d5fe-4a08-a4e4-180c1c3c5972","Type":"ContainerDied","Data":"34872d8e140d477d4e1acaff2892d5365efd05027a5e8755d308114fc0503dae"} Oct 31 20:05:46.372603 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.372582 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn" event={"ID":"a673180d-8ebc-474d-a6d1-130ea7603aed","Type":"ContainerDied","Data":"e89e0ccf109a15076d698512112188c3e3db905b4fe724acadb78a581ced39fc"} Oct 31 20:05:46.372759 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.372744 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e89e0ccf109a15076d698512112188c3e3db905b4fe724acadb78a581ced39fc" Oct 31 20:05:46.372960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.372947 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn" Oct 31 20:05:46.447950 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.447926 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 20:05:46.490034 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.490001 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 20:05:46.498322 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.498292 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-data\") pod \"949b61a5-f07e-47c1-b360-a3bfde012f42\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " Oct 31 20:05:46.498795 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.498776 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-token\" (UniqueName: \"kubernetes.io/projected/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-token\") pod \"949b61a5-f07e-47c1-b360-a3bfde012f42\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " Oct 31 20:05:46.498960 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.498725 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-data" pod "949b61a5-f07e-47c1-b360-a3bfde012f42" (UID: "949b61a5-f07e-47c1-b360-a3bfde012f42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:46.499034 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.498958 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-workload-socket\") pod \"949b61a5-f07e-47c1-b360-a3bfde012f42\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " Oct 31 20:05:46.499034 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.498991 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-credential-socket\") pod \"949b61a5-f07e-47c1-b360-a3bfde012f42\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " Oct 31 20:05:46.499148 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.499036 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-workload-certs\") pod \"949b61a5-f07e-47c1-b360-a3bfde012f42\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " Oct 31 20:05:46.499278 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.499234 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-workload-socket" pod "949b61a5-f07e-47c1-b360-a3bfde012f42" (UID: "949b61a5-f07e-47c1-b360-a3bfde012f42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:46.499352 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.499294 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-credential-socket" pod "949b61a5-f07e-47c1-b360-a3bfde012f42" (UID: "949b61a5-f07e-47c1-b360-a3bfde012f42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:46.499399 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.499355 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-workload-certs" pod "949b61a5-f07e-47c1-b360-a3bfde012f42" (UID: "949b61a5-f07e-47c1-b360-a3bfde012f42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:46.499438 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.499399 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/949b61a5-f07e-47c1-b360-a3bfde012f42-kube-api-access-8zz8n\" (UniqueName: \"kubernetes.io/projected/949b61a5-f07e-47c1-b360-a3bfde012f42-kube-api-access-8zz8n\") pod \"949b61a5-f07e-47c1-b360-a3bfde012f42\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " Oct 31 20:05:46.499544 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.499528 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-envoy\") pod \"949b61a5-f07e-47c1-b360-a3bfde012f42\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " Oct 31 20:05:46.499629 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.499613 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/949b61a5-f07e-47c1-b360-a3bfde012f42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/949b61a5-f07e-47c1-b360-a3bfde012f42-istiod-ca-cert\") pod \"949b61a5-f07e-47c1-b360-a3bfde012f42\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " Oct 31 20:05:46.499756 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.499656 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-podinfo\") pod \"949b61a5-f07e-47c1-b360-a3bfde012f42\" (UID: \"949b61a5-f07e-47c1-b360-a3bfde012f42\") " Oct 31 20:05:46.500004 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.499980 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/949b61a5-f07e-47c1-b360-a3bfde012f42-istiod-ca-cert" pod "949b61a5-f07e-47c1-b360-a3bfde012f42" (UID: "949b61a5-f07e-47c1-b360-a3bfde012f42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:05:46.500229 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.500202 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/949b61a5-f07e-47c1-b360-a3bfde012f42-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.500313 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.500232 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.500313 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.500248 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.500313 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.500262 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.500313 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.500275 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.502219 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.502193 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-podinfo" pod "949b61a5-f07e-47c1-b360-a3bfde012f42" (UID: "949b61a5-f07e-47c1-b360-a3bfde012f42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:05:46.502746 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.502726 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/949b61a5-f07e-47c1-b360-a3bfde012f42-kube-api-access-8zz8n" pod "949b61a5-f07e-47c1-b360-a3bfde012f42" (UID: "949b61a5-f07e-47c1-b360-a3bfde012f42"). InnerVolumeSpecName "kube-api-access-8zz8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:46.503947 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.503902 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-token" pod "949b61a5-f07e-47c1-b360-a3bfde012f42" (UID: "949b61a5-f07e-47c1-b360-a3bfde012f42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:46.504836 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.504811 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-envoy" pod "949b61a5-f07e-47c1-b360-a3bfde012f42" (UID: "949b61a5-f07e-47c1-b360-a3bfde012f42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:46.546329 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.546294 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 20:05:46.553186 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.552929 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8"] Oct 31 20:05:46.563179 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.563151 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz"] Oct 31 20:05:46.601365 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.601328 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-token\" (UniqueName: \"kubernetes.io/projected/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-token\") pod \"ceb11f75-3d05-4e9c-8db7-caa224621cad\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " Oct 31 20:05:46.601526 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.601399 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-credential-socket\") pod \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " Oct 31 20:05:46.601526 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.601455 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-workload-certs\") pod \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " Oct 31 20:05:46.601601 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.601525 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-token\" (UniqueName: \"kubernetes.io/projected/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-token\") pod \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " Oct 31 20:05:46.601601 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.601555 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istiod-ca-cert\") pod \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " Oct 31 20:05:46.601601 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.601588 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceb11f75-3d05-4e9c-8db7-caa224621cad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb11f75-3d05-4e9c-8db7-caa224621cad-istiod-ca-cert\") pod \"ceb11f75-3d05-4e9c-8db7-caa224621cad\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " Oct 31 20:05:46.601701 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.601628 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-envoy\") pod \"ceb11f75-3d05-4e9c-8db7-caa224621cad\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " Oct 31 20:05:46.601736 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.601697 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-podinfo\") pod \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " Oct 31 20:05:46.601789 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.601763 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-data\") pod \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " Oct 31 20:05:46.601831 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.601815 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-data\") pod \"ceb11f75-3d05-4e9c-8db7-caa224621cad\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " Oct 31 20:05:46.601898 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.601877 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-envoy\") pod \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " Oct 31 20:05:46.602124 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.602098 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceb11f75-3d05-4e9c-8db7-caa224621cad-kube-api-access-xk2v4\" (UniqueName: \"kubernetes.io/projected/ceb11f75-3d05-4e9c-8db7-caa224621cad-kube-api-access-xk2v4\") pod \"ceb11f75-3d05-4e9c-8db7-caa224621cad\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " Oct 31 20:05:46.602204 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.602164 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-credential-socket\") pod \"ceb11f75-3d05-4e9c-8db7-caa224621cad\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " Oct 31 20:05:46.602239 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.602221 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-workload-socket\") pod \"ceb11f75-3d05-4e9c-8db7-caa224621cad\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " Oct 31 20:05:46.602317 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.602299 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-workload-socket\") pod \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " Oct 31 20:05:46.602478 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.602460 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-kube-api-access-j8h26\" (UniqueName: \"kubernetes.io/projected/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-kube-api-access-j8h26\") pod \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\" (UID: \"f41d2753-d5fe-4a08-a4e4-180c1c3c5972\") " Oct 31 20:05:46.602828 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.602810 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-workload-certs\") pod \"ceb11f75-3d05-4e9c-8db7-caa224621cad\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " Oct 31 20:05:46.604443 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.604424 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-podinfo\") pod \"ceb11f75-3d05-4e9c-8db7-caa224621cad\" (UID: \"ceb11f75-3d05-4e9c-8db7-caa224621cad\") " Oct 31 20:05:46.606299 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.602338 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istiod-ca-cert" pod "f41d2753-d5fe-4a08-a4e4-180c1c3c5972" (UID: "f41d2753-d5fe-4a08-a4e4-180c1c3c5972"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:05:46.606299 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.602480 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-data" pod "ceb11f75-3d05-4e9c-8db7-caa224621cad" (UID: "ceb11f75-3d05-4e9c-8db7-caa224621cad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:46.606299 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.602704 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-credential-socket" pod "ceb11f75-3d05-4e9c-8db7-caa224621cad" (UID: "ceb11f75-3d05-4e9c-8db7-caa224621cad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:46.606299 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.602724 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceb11f75-3d05-4e9c-8db7-caa224621cad-istiod-ca-cert" pod "ceb11f75-3d05-4e9c-8db7-caa224621cad" (UID: "ceb11f75-3d05-4e9c-8db7-caa224621cad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:05:46.606299 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.603612 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-credential-socket" pod "f41d2753-d5fe-4a08-a4e4-180c1c3c5972" (UID: "f41d2753-d5fe-4a08-a4e4-180c1c3c5972"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:46.606299 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.603898 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-workload-socket" pod "f41d2753-d5fe-4a08-a4e4-180c1c3c5972" (UID: "f41d2753-d5fe-4a08-a4e4-180c1c3c5972"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:46.606299 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.604117 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-workload-socket" pod "ceb11f75-3d05-4e9c-8db7-caa224621cad" (UID: "ceb11f75-3d05-4e9c-8db7-caa224621cad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:46.606299 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.604666 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-workload-certs" pod "f41d2753-d5fe-4a08-a4e4-180c1c3c5972" (UID: "f41d2753-d5fe-4a08-a4e4-180c1c3c5972"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:46.606299 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.604847 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-token" pod "ceb11f75-3d05-4e9c-8db7-caa224621cad" (UID: "ceb11f75-3d05-4e9c-8db7-caa224621cad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:46.606299 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.605401 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-workload-certs" pod "ceb11f75-3d05-4e9c-8db7-caa224621cad" (UID: "ceb11f75-3d05-4e9c-8db7-caa224621cad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:46.606299 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.606130 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-podinfo" pod "f41d2753-d5fe-4a08-a4e4-180c1c3c5972" (UID: "f41d2753-d5fe-4a08-a4e4-180c1c3c5972"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:05:46.606299 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.606195 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-envoy" pod "ceb11f75-3d05-4e9c-8db7-caa224621cad" (UID: "ceb11f75-3d05-4e9c-8db7-caa224621cad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:46.607148 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.606672 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-kube-api-access-j8h26" pod "f41d2753-d5fe-4a08-a4e4-180c1c3c5972" (UID: "f41d2753-d5fe-4a08-a4e4-180c1c3c5972"). InnerVolumeSpecName "kube-api-access-j8h26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:46.607148 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.607081 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb11f75-3d05-4e9c-8db7-caa224621cad-kube-api-access-xk2v4" pod "ceb11f75-3d05-4e9c-8db7-caa224621cad" (UID: "ceb11f75-3d05-4e9c-8db7-caa224621cad"). InnerVolumeSpecName "kube-api-access-xk2v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:46.607372 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.607334 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-data" pod "f41d2753-d5fe-4a08-a4e4-180c1c3c5972" (UID: "f41d2753-d5fe-4a08-a4e4-180c1c3c5972"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:46.607787 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.607703 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-token" pod "f41d2753-d5fe-4a08-a4e4-180c1c3c5972" (UID: "f41d2753-d5fe-4a08-a4e4-180c1c3c5972"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:46.607884 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.607847 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.607884 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.607873 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.608103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.607888 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zz8n\" (UniqueName: \"kubernetes.io/projected/949b61a5-f07e-47c1-b360-a3bfde012f42-kube-api-access-8zz8n\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.608103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.607903 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.608103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.607939 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.608103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.607952 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb11f75-3d05-4e9c-8db7-caa224621cad-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.608103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.607966 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.608103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.607980 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.608103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.607995 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.608103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.608008 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.608103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.608021 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.608103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.608033 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk2v4\" (UniqueName: \"kubernetes.io/projected/ceb11f75-3d05-4e9c-8db7-caa224621cad-kube-api-access-xk2v4\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.608103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.608047 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.608103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.608061 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.608103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.608074 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/949b61a5-f07e-47c1-b360-a3bfde012f42-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.608103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.608088 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.608103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.608101 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.608900 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.608115 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8h26\" (UniqueName: \"kubernetes.io/projected/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-kube-api-access-j8h26\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.608900 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.608128 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb11f75-3d05-4e9c-8db7-caa224621cad-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.608900 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.608142 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.609186 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.609020 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-podinfo" pod "ceb11f75-3d05-4e9c-8db7-caa224621cad" (UID: "ceb11f75-3d05-4e9c-8db7-caa224621cad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:05:46.609186 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.609028 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-envoy" pod "f41d2753-d5fe-4a08-a4e4-180c1c3c5972" (UID: "f41d2753-d5fe-4a08-a4e4-180c1c3c5972"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:46.708922 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.708881 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f41d2753-d5fe-4a08-a4e4-180c1c3c5972-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:46.708922 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:46.708929 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb11f75-3d05-4e9c-8db7-caa224621cad-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:47.381975 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:47.381943 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" Oct 31 20:05:47.381975 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:47.381952 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr" event={"ID":"ceb11f75-3d05-4e9c-8db7-caa224621cad","Type":"ContainerDied","Data":"021584c5023e4f58c7d21eba4fe24ae32d43359747184e9e1484be3ea735481d"} Oct 31 20:05:47.382511 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:47.382031 4910 scope.go:122] "RemoveContainer" containerID="fedbdb204387b3305e30411da5ecfa09c5282cfb45d84d787e225acd28eac397" Oct 31 20:05:47.383795 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:47.383774 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" Oct 31 20:05:47.383949 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:47.383813 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2" event={"ID":"949b61a5-f07e-47c1-b360-a3bfde012f42","Type":"ContainerDied","Data":"cb27839590dea226c7fb3b7ea54a3d222636adda71df82a43c0c2b02b5909f7e"} Oct 31 20:05:47.386575 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:47.386536 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" event={"ID":"f41d2753-d5fe-4a08-a4e4-180c1c3c5972","Type":"ContainerDied","Data":"e3f9fd8374eb4ae15c72202497b4961e85996e67cd72238980bfea39637386c4"} Oct 31 20:05:47.386680 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:47.386631 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm" Oct 31 20:05:47.395560 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:47.395535 4910 scope.go:122] "RemoveContainer" containerID="771ee87af4ec83f3266431bf0447e56a1d0d18820ff12724e8d48d0c100dabac" Oct 31 20:05:47.407043 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:47.407023 4910 scope.go:122] "RemoveContainer" containerID="34872d8e140d477d4e1acaff2892d5365efd05027a5e8755d308114fc0503dae" Oct 31 20:05:47.436645 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:47.436618 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr"] Oct 31 20:05:47.445204 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:47.445176 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-sb6nr"] Oct 31 20:05:47.469317 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:47.469228 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm"] Oct 31 20:05:47.474282 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:47.474258 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-jnkxm"] Oct 31 20:05:47.488332 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:47.488308 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2"] Oct 31 20:05:47.493742 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:47.493716 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-x5jh2"] Oct 31 20:05:47.758411 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:47.758317 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="949b61a5-f07e-47c1-b360-a3bfde012f42" path="/var/lib/kubelet/pods/949b61a5-f07e-47c1-b360-a3bfde012f42/volumes" Oct 31 20:05:47.759126 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:47.759105 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceb11f75-3d05-4e9c-8db7-caa224621cad" path="/var/lib/kubelet/pods/ceb11f75-3d05-4e9c-8db7-caa224621cad/volumes" Oct 31 20:05:47.759763 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:47.759745 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f41d2753-d5fe-4a08-a4e4-180c1c3c5972" path="/var/lib/kubelet/pods/f41d2753-d5fe-4a08-a4e4-180c1c3c5972/volumes" Oct 31 20:05:49.375624 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:49.375588 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs"] Oct 31 20:05:49.376470 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:49.376407 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" podUID="204c09e1-cfce-4add-986d-165e78767839" containerName="istio-proxy" containerID="cri-o://88ba2c1333b4b3515b53ad6303bc2ea7ab6becea3f4431c72fd65d3472629b76" gracePeriod=30 Oct 31 20:05:50.100935 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:50.100875 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq"] Oct 31 20:05:50.776190 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:50.776149 4910 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:05:50.776597 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:50.776214 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" podUID="204c09e1-cfce-4add-986d-165e78767839" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:05:51.900439 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:51.900403 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9"] Oct 31 20:05:52.479868 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.479835 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee17cfdc-nrwsf"] Oct 31 20:05:52.480466 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.480444 4910 cpu_manager.go:414] "containerMap: removing container" podUID="ceb11f75-3d05-4e9c-8db7-caa224621cad" containerName="istio-proxy" Oct 31 20:05:52.480571 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.480467 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceb11f75-3d05-4e9c-8db7-caa224621cad" containerName="istio-proxy" Oct 31 20:05:52.480571 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.480489 4910 cpu_manager.go:414] "containerMap: removing container" podUID="949b61a5-f07e-47c1-b360-a3bfde012f42" containerName="istio-proxy" Oct 31 20:05:52.480571 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.480496 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="949b61a5-f07e-47c1-b360-a3bfde012f42" containerName="istio-proxy" Oct 31 20:05:52.480571 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.480518 4910 cpu_manager.go:414] "containerMap: removing container" podUID="e898f0b3-1c73-44d5-bc56-1d2c5673d516" containerName="istio-proxy" Oct 31 20:05:52.480571 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.480526 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e898f0b3-1c73-44d5-bc56-1d2c5673d516" containerName="istio-proxy" Oct 31 20:05:52.480571 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.480550 4910 cpu_manager.go:414] "containerMap: removing container" podUID="a673180d-8ebc-474d-a6d1-130ea7603aed" containerName="revoke-keys" Oct 31 20:05:52.480571 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.480558 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a673180d-8ebc-474d-a6d1-130ea7603aed" containerName="revoke-keys" Oct 31 20:05:52.480948 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.480575 4910 cpu_manager.go:414] "containerMap: removing container" podUID="f41d2753-d5fe-4a08-a4e4-180c1c3c5972" containerName="istio-proxy" Oct 31 20:05:52.480948 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.480583 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f41d2753-d5fe-4a08-a4e4-180c1c3c5972" containerName="istio-proxy" Oct 31 20:05:52.480948 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.480599 4910 cpu_manager.go:414] "containerMap: removing container" podUID="949b61a5-f07e-47c1-b360-a3bfde012f42" containerName="istio-proxy" Oct 31 20:05:52.480948 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.480607 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="949b61a5-f07e-47c1-b360-a3bfde012f42" containerName="istio-proxy" Oct 31 20:05:52.480948 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.480770 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="949b61a5-f07e-47c1-b360-a3bfde012f42" containerName="istio-proxy" Oct 31 20:05:52.480948 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.480784 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="e898f0b3-1c73-44d5-bc56-1d2c5673d516" containerName="istio-proxy" Oct 31 20:05:52.480948 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.480798 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="a673180d-8ebc-474d-a6d1-130ea7603aed" containerName="revoke-keys" Oct 31 20:05:52.480948 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.480810 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="f41d2753-d5fe-4a08-a4e4-180c1c3c5972" containerName="istio-proxy" Oct 31 20:05:52.480948 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.480825 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceb11f75-3d05-4e9c-8db7-caa224621cad" containerName="istio-proxy" Oct 31 20:05:52.486498 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.486474 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee17cfdc-nrwsf" Oct 31 20:05:52.498305 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.498264 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee17cfdc-nrwsf"] Oct 31 20:05:52.576103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.576061 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:05:52.576103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.576082 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 20:05:52.605042 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.605004 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901b6b03-183b-4819-a239-7ae5565e9eba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee17cfdc-nrwsf\" (UID: \"901b6b03-183b-4819-a239-7ae5565e9eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee17cfdc-nrwsf" Oct 31 20:05:52.605221 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.605140 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc5vp\" (UniqueName: \"kubernetes.io/projected/901b6b03-183b-4819-a239-7ae5565e9eba-kube-api-access-dc5vp\") pod \"maas-api-revoke-sub-ee17cfdc-nrwsf\" (UID: \"901b6b03-183b-4819-a239-7ae5565e9eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee17cfdc-nrwsf" Oct 31 20:05:52.605269 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.605233 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901b6b03-183b-4819-a239-7ae5565e9eba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee17cfdc-nrwsf\" (UID: \"901b6b03-183b-4819-a239-7ae5565e9eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee17cfdc-nrwsf" Oct 31 20:05:52.706775 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.706732 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901b6b03-183b-4819-a239-7ae5565e9eba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee17cfdc-nrwsf\" (UID: \"901b6b03-183b-4819-a239-7ae5565e9eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee17cfdc-nrwsf" Oct 31 20:05:52.707029 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.706894 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901b6b03-183b-4819-a239-7ae5565e9eba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee17cfdc-nrwsf\" (UID: \"901b6b03-183b-4819-a239-7ae5565e9eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee17cfdc-nrwsf" Oct 31 20:05:52.707029 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.707019 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc5vp\" (UniqueName: \"kubernetes.io/projected/901b6b03-183b-4819-a239-7ae5565e9eba-kube-api-access-dc5vp\") pod \"maas-api-revoke-sub-ee17cfdc-nrwsf\" (UID: \"901b6b03-183b-4819-a239-7ae5565e9eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee17cfdc-nrwsf" Oct 31 20:05:52.707604 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.707579 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901b6b03-183b-4819-a239-7ae5565e9eba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee17cfdc-nrwsf\" (UID: \"901b6b03-183b-4819-a239-7ae5565e9eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee17cfdc-nrwsf" Oct 31 20:05:52.709176 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.709155 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901b6b03-183b-4819-a239-7ae5565e9eba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee17cfdc-nrwsf\" (UID: \"901b6b03-183b-4819-a239-7ae5565e9eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee17cfdc-nrwsf" Oct 31 20:05:52.721518 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.721493 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc5vp\" (UniqueName: \"kubernetes.io/projected/901b6b03-183b-4819-a239-7ae5565e9eba-kube-api-access-dc5vp\") pod \"maas-api-revoke-sub-ee17cfdc-nrwsf\" (UID: \"901b6b03-183b-4819-a239-7ae5565e9eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee17cfdc-nrwsf" Oct 31 20:05:52.819386 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:52.818203 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee17cfdc-nrwsf" Oct 31 20:05:53.125244 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:05:53.125207 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod901b6b03_183b_4819_a239_7ae5565e9eba.slice/crio-2022c4bc49514bbda5760dc60b2b02d2529848b0faeb6addb15736f3ffe7b929 WatchSource:0}: Error finding container 2022c4bc49514bbda5760dc60b2b02d2529848b0faeb6addb15736f3ffe7b929: Status 404 returned error can't find the container with id 2022c4bc49514bbda5760dc60b2b02d2529848b0faeb6addb15736f3ffe7b929 Oct 31 20:05:53.127052 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:53.127026 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee17cfdc-nrwsf"] Oct 31 20:05:53.481266 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:53.481219 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee17cfdc-nrwsf" event={"ID":"901b6b03-183b-4819-a239-7ae5565e9eba","Type":"ContainerStarted","Data":"2022c4bc49514bbda5760dc60b2b02d2529848b0faeb6addb15736f3ffe7b929"} Oct 31 20:05:54.491033 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.490998 4910 generic.go:356] "Generic (PLEG): container finished" podID="901b6b03-183b-4819-a239-7ae5565e9eba" containerID="090386b2ee3d22eea123a6347fb90d6a13a4a1c5c7565db075c684b33a288715" exitCode=0 Oct 31 20:05:54.491531 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.491074 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee17cfdc-nrwsf" event={"ID":"901b6b03-183b-4819-a239-7ae5565e9eba","Type":"ContainerDied","Data":"090386b2ee3d22eea123a6347fb90d6a13a4a1c5c7565db075c684b33a288715"} Oct 31 20:05:54.705936 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.705893 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 20:05:54.835690 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.835657 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/204c09e1-cfce-4add-986d-165e78767839-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204c09e1-cfce-4add-986d-165e78767839-istio-podinfo\") pod \"204c09e1-cfce-4add-986d-165e78767839\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " Oct 31 20:05:54.835846 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.835705 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-credential-socket\") pod \"204c09e1-cfce-4add-986d-165e78767839\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " Oct 31 20:05:54.835846 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.835742 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-istio-envoy\") pod \"204c09e1-cfce-4add-986d-165e78767839\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " Oct 31 20:05:54.835968 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.835857 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-istio-data\") pod \"204c09e1-cfce-4add-986d-165e78767839\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " Oct 31 20:05:54.835968 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.835960 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/204c09e1-cfce-4add-986d-165e78767839-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204c09e1-cfce-4add-986d-165e78767839-istiod-ca-cert\") pod \"204c09e1-cfce-4add-986d-165e78767839\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " Oct 31 20:05:54.836055 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.835999 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-workload-certs\") pod \"204c09e1-cfce-4add-986d-165e78767839\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " Oct 31 20:05:54.836055 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.836042 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204c09e1-cfce-4add-986d-165e78767839-kube-api-access-4k7zd\" (UniqueName: \"kubernetes.io/projected/204c09e1-cfce-4add-986d-165e78767839-kube-api-access-4k7zd\") pod \"204c09e1-cfce-4add-986d-165e78767839\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " Oct 31 20:05:54.836241 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.836084 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204c09e1-cfce-4add-986d-165e78767839-istio-token\" (UniqueName: \"kubernetes.io/projected/204c09e1-cfce-4add-986d-165e78767839-istio-token\") pod \"204c09e1-cfce-4add-986d-165e78767839\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " Oct 31 20:05:54.836241 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.836077 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-credential-socket" pod "204c09e1-cfce-4add-986d-165e78767839" (UID: "204c09e1-cfce-4add-986d-165e78767839"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:54.836241 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.836162 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-workload-socket\") pod \"204c09e1-cfce-4add-986d-165e78767839\" (UID: \"204c09e1-cfce-4add-986d-165e78767839\") " Oct 31 20:05:54.836444 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.836421 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204c09e1-cfce-4add-986d-165e78767839-istiod-ca-cert" pod "204c09e1-cfce-4add-986d-165e78767839" (UID: "204c09e1-cfce-4add-986d-165e78767839"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:05:54.836654 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.836627 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-workload-certs" pod "204c09e1-cfce-4add-986d-165e78767839" (UID: "204c09e1-cfce-4add-986d-165e78767839"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:54.836736 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.836659 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-workload-socket" pod "204c09e1-cfce-4add-986d-165e78767839" (UID: "204c09e1-cfce-4add-986d-165e78767839"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:54.836808 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.836790 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-istio-data" pod "204c09e1-cfce-4add-986d-165e78767839" (UID: "204c09e1-cfce-4add-986d-165e78767839"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:54.836948 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.836933 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204c09e1-cfce-4add-986d-165e78767839-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:54.836989 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.836954 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:54.836989 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.836969 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:54.836989 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.836983 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:54.837123 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.836996 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:54.838186 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.838161 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-istio-envoy" pod "204c09e1-cfce-4add-986d-165e78767839" (UID: "204c09e1-cfce-4add-986d-165e78767839"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:05:54.838523 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.838500 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/204c09e1-cfce-4add-986d-165e78767839-istio-podinfo" pod "204c09e1-cfce-4add-986d-165e78767839" (UID: "204c09e1-cfce-4add-986d-165e78767839"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:05:54.838523 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.838505 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204c09e1-cfce-4add-986d-165e78767839-istio-token" pod "204c09e1-cfce-4add-986d-165e78767839" (UID: "204c09e1-cfce-4add-986d-165e78767839"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:54.838806 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.838779 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204c09e1-cfce-4add-986d-165e78767839-kube-api-access-4k7zd" pod "204c09e1-cfce-4add-986d-165e78767839" (UID: "204c09e1-cfce-4add-986d-165e78767839"). InnerVolumeSpecName "kube-api-access-4k7zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:54.938382 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.938308 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204c09e1-cfce-4add-986d-165e78767839-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:54.938382 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.938336 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204c09e1-cfce-4add-986d-165e78767839-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:54.938382 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.938345 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k7zd\" (UniqueName: \"kubernetes.io/projected/204c09e1-cfce-4add-986d-165e78767839-kube-api-access-4k7zd\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:54.938382 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:54.938356 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204c09e1-cfce-4add-986d-165e78767839-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:55.102213 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:55.102150 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq" podUID="7d56bb36-ed0e-4127-9461-cbec40a46b95" containerName="payload-processing" containerID="cri-o://b9e5a644cfb960de024dd921569b10f28f4c48cc6a8c226291b4a417e6ee3b35" gracePeriod=25 Oct 31 20:05:55.503520 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:55.503488 4910 generic.go:356] "Generic (PLEG): container finished" podID="204c09e1-cfce-4add-986d-165e78767839" containerID="88ba2c1333b4b3515b53ad6303bc2ea7ab6becea3f4431c72fd65d3472629b76" exitCode=0 Oct 31 20:05:55.503995 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:55.503539 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" Oct 31 20:05:55.503995 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:55.503567 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" event={"ID":"204c09e1-cfce-4add-986d-165e78767839","Type":"ContainerDied","Data":"88ba2c1333b4b3515b53ad6303bc2ea7ab6becea3f4431c72fd65d3472629b76"} Oct 31 20:05:55.503995 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:55.503620 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs" event={"ID":"204c09e1-cfce-4add-986d-165e78767839","Type":"ContainerDied","Data":"7eebae0b0db0ac4b03e40f4ffb0f3368042567cb7d80d84d6a31c9db2e8a30c8"} Oct 31 20:05:55.503995 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:55.503642 4910 scope.go:122] "RemoveContainer" containerID="88ba2c1333b4b3515b53ad6303bc2ea7ab6becea3f4431c72fd65d3472629b76" Oct 31 20:05:55.515897 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:55.515871 4910 scope.go:122] "RemoveContainer" containerID="de598823f128e0f7ff708756df6ed86a22abc66e59338acd2711304ab9b4a257" Oct 31 20:05:55.528999 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:55.528932 4910 scope.go:122] "RemoveContainer" containerID="88ba2c1333b4b3515b53ad6303bc2ea7ab6becea3f4431c72fd65d3472629b76" Oct 31 20:05:55.529322 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:05:55.529295 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ba2c1333b4b3515b53ad6303bc2ea7ab6becea3f4431c72fd65d3472629b76\": container with ID starting with 88ba2c1333b4b3515b53ad6303bc2ea7ab6becea3f4431c72fd65d3472629b76 not found: ID does not exist" containerID="88ba2c1333b4b3515b53ad6303bc2ea7ab6becea3f4431c72fd65d3472629b76" Oct 31 20:05:55.529438 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:55.529334 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ba2c1333b4b3515b53ad6303bc2ea7ab6becea3f4431c72fd65d3472629b76"} err="failed to get container status \"88ba2c1333b4b3515b53ad6303bc2ea7ab6becea3f4431c72fd65d3472629b76\": rpc error: code = NotFound desc = could not find container \"88ba2c1333b4b3515b53ad6303bc2ea7ab6becea3f4431c72fd65d3472629b76\": container with ID starting with 88ba2c1333b4b3515b53ad6303bc2ea7ab6becea3f4431c72fd65d3472629b76 not found: ID does not exist" Oct 31 20:05:55.529438 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:55.529358 4910 scope.go:122] "RemoveContainer" containerID="de598823f128e0f7ff708756df6ed86a22abc66e59338acd2711304ab9b4a257" Oct 31 20:05:55.529656 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:05:55.529631 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de598823f128e0f7ff708756df6ed86a22abc66e59338acd2711304ab9b4a257\": container with ID starting with de598823f128e0f7ff708756df6ed86a22abc66e59338acd2711304ab9b4a257 not found: ID does not exist" containerID="de598823f128e0f7ff708756df6ed86a22abc66e59338acd2711304ab9b4a257" Oct 31 20:05:55.529739 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:55.529668 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de598823f128e0f7ff708756df6ed86a22abc66e59338acd2711304ab9b4a257"} err="failed to get container status \"de598823f128e0f7ff708756df6ed86a22abc66e59338acd2711304ab9b4a257\": rpc error: code = NotFound desc = could not find container \"de598823f128e0f7ff708756df6ed86a22abc66e59338acd2711304ab9b4a257\": container with ID starting with de598823f128e0f7ff708756df6ed86a22abc66e59338acd2711304ab9b4a257 not found: ID does not exist" Oct 31 20:05:55.538556 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:55.538530 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs"] Oct 31 20:05:55.547795 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:55.547766 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-g9hvs"] Oct 31 20:05:55.707007 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:55.706982 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee17cfdc-nrwsf" Oct 31 20:05:55.756868 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:55.756782 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="204c09e1-cfce-4add-986d-165e78767839" path="/var/lib/kubelet/pods/204c09e1-cfce-4add-986d-165e78767839/volumes" Oct 31 20:05:55.761900 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:55.761864 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/901b6b03-183b-4819-a239-7ae5565e9eba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901b6b03-183b-4819-a239-7ae5565e9eba-maas-api-openshift-ca\") pod \"901b6b03-183b-4819-a239-7ae5565e9eba\" (UID: \"901b6b03-183b-4819-a239-7ae5565e9eba\") " Oct 31 20:05:55.762159 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:55.761986 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/901b6b03-183b-4819-a239-7ae5565e9eba-kube-api-access-dc5vp\" (UniqueName: \"kubernetes.io/projected/901b6b03-183b-4819-a239-7ae5565e9eba-kube-api-access-dc5vp\") pod \"901b6b03-183b-4819-a239-7ae5565e9eba\" (UID: \"901b6b03-183b-4819-a239-7ae5565e9eba\") " Oct 31 20:05:55.762159 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:55.762021 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/901b6b03-183b-4819-a239-7ae5565e9eba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901b6b03-183b-4819-a239-7ae5565e9eba-maas-api-xks-ca\") pod \"901b6b03-183b-4819-a239-7ae5565e9eba\" (UID: \"901b6b03-183b-4819-a239-7ae5565e9eba\") " Oct 31 20:05:55.763504 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:55.763472 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/901b6b03-183b-4819-a239-7ae5565e9eba-maas-api-openshift-ca" pod "901b6b03-183b-4819-a239-7ae5565e9eba" (UID: "901b6b03-183b-4819-a239-7ae5565e9eba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:05:55.770003 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:55.765899 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/901b6b03-183b-4819-a239-7ae5565e9eba-kube-api-access-dc5vp" pod "901b6b03-183b-4819-a239-7ae5565e9eba" (UID: "901b6b03-183b-4819-a239-7ae5565e9eba"). InnerVolumeSpecName "kube-api-access-dc5vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:05:55.770340 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:55.770261 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/901b6b03-183b-4819-a239-7ae5565e9eba-maas-api-xks-ca" pod "901b6b03-183b-4819-a239-7ae5565e9eba" (UID: "901b6b03-183b-4819-a239-7ae5565e9eba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:05:55.864131 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:55.864094 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc5vp\" (UniqueName: \"kubernetes.io/projected/901b6b03-183b-4819-a239-7ae5565e9eba-kube-api-access-dc5vp\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:55.864131 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:55.864130 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/901b6b03-183b-4819-a239-7ae5565e9eba-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:55.864353 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:55.864140 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/901b6b03-183b-4819-a239-7ae5565e9eba-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:05:56.512353 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:56.512322 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee17cfdc-nrwsf" Oct 31 20:05:56.512841 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:56.512357 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee17cfdc-nrwsf" event={"ID":"901b6b03-183b-4819-a239-7ae5565e9eba","Type":"ContainerDied","Data":"2022c4bc49514bbda5760dc60b2b02d2529848b0faeb6addb15736f3ffe7b929"} Oct 31 20:05:56.512841 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:56.512401 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2022c4bc49514bbda5760dc60b2b02d2529848b0faeb6addb15736f3ffe7b929" Oct 31 20:05:56.901652 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:56.901610 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9" podUID="c34f06bd-3b50-4342-9719-bcccd4be955f" containerName="payload-processing" containerID="cri-o://938d28e68583a4153252ecef43fbb6e4c1e53561bcb89640f939bc7a91e6d39c" gracePeriod=25 Oct 31 20:05:59.648035 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:59.647957 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" containerID="cri-o://6d3c3829dd2141679e7092f74dd2f4742aa00d1440e5d42dd81ead069e4465fe" gracePeriod=45 Oct 31 20:05:59.700952 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:05:59.700901 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" containerID="cri-o://c1eba0f875ec3982501b1251bc2c1fa1b6980d1ffe1cfb6e0b18def31e13e8de" gracePeriod=45 Oct 31 20:06:00.091042 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.090617 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 20:06:00.225718 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.225686 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 20:06:00.240579 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.240544 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77506413-2985-4c17-b98c-dc414769ec5d-tls-certs\" (UniqueName: \"kubernetes.io/secret/77506413-2985-4c17-b98c-dc414769ec5d-tls-certs\") pod \"77506413-2985-4c17-b98c-dc414769ec5d\" (UID: \"77506413-2985-4c17-b98c-dc414769ec5d\") " Oct 31 20:06:00.240772 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.240609 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77506413-2985-4c17-b98c-dc414769ec5d-kube-api-access-wk5wh\" (UniqueName: \"kubernetes.io/projected/77506413-2985-4c17-b98c-dc414769ec5d-kube-api-access-wk5wh\") pod \"77506413-2985-4c17-b98c-dc414769ec5d\" (UID: \"77506413-2985-4c17-b98c-dc414769ec5d\") " Oct 31 20:06:00.240772 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.240697 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-tmp-dir\") pod \"77506413-2985-4c17-b98c-dc414769ec5d\" (UID: \"77506413-2985-4c17-b98c-dc414769ec5d\") " Oct 31 20:06:00.240772 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.240732 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-model-cache\") pod \"77506413-2985-4c17-b98c-dc414769ec5d\" (UID: \"77506413-2985-4c17-b98c-dc414769ec5d\") " Oct 31 20:06:00.240772 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.240752 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-home\" (UniqueName: \"kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-home\") pod \"77506413-2985-4c17-b98c-dc414769ec5d\" (UID: \"77506413-2985-4c17-b98c-dc414769ec5d\") " Oct 31 20:06:00.241017 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.240867 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-dshm\") pod \"77506413-2985-4c17-b98c-dc414769ec5d\" (UID: \"77506413-2985-4c17-b98c-dc414769ec5d\") " Oct 31 20:06:00.241318 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.241284 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-tmp-dir" pod "77506413-2985-4c17-b98c-dc414769ec5d" (UID: "77506413-2985-4c17-b98c-dc414769ec5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:00.242211 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.241309 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-home" pod "77506413-2985-4c17-b98c-dc414769ec5d" (UID: "77506413-2985-4c17-b98c-dc414769ec5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:00.242211 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.241476 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-tmp-dir\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:00.242211 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.241495 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-home\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:00.242211 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.241530 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-model-cache" pod "77506413-2985-4c17-b98c-dc414769ec5d" (UID: "77506413-2985-4c17-b98c-dc414769ec5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:00.245322 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.245301 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77506413-2985-4c17-b98c-dc414769ec5d-tls-certs" pod "77506413-2985-4c17-b98c-dc414769ec5d" (UID: "77506413-2985-4c17-b98c-dc414769ec5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:06:00.245479 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.245309 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77506413-2985-4c17-b98c-dc414769ec5d-kube-api-access-wk5wh" pod "77506413-2985-4c17-b98c-dc414769ec5d" (UID: "77506413-2985-4c17-b98c-dc414769ec5d"). InnerVolumeSpecName "kube-api-access-wk5wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:06:00.246641 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.246615 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-dshm" pod "77506413-2985-4c17-b98c-dc414769ec5d" (UID: "77506413-2985-4c17-b98c-dc414769ec5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:00.316523 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.316486 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr"] Oct 31 20:06:00.348951 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.343479 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-dshm\") pod \"6db020e0-c316-47b9-ad20-a920d75f6138\" (UID: \"6db020e0-c316-47b9-ad20-a920d75f6138\") " Oct 31 20:06:00.348951 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.343610 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6db020e0-c316-47b9-ad20-a920d75f6138-tls-certs\" (UniqueName: \"kubernetes.io/secret/6db020e0-c316-47b9-ad20-a920d75f6138-tls-certs\") pod \"6db020e0-c316-47b9-ad20-a920d75f6138\" (UID: \"6db020e0-c316-47b9-ad20-a920d75f6138\") " Oct 31 20:06:00.348951 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.343669 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db020e0-c316-47b9-ad20-a920d75f6138-kube-api-access-txh8f\" (UniqueName: \"kubernetes.io/projected/6db020e0-c316-47b9-ad20-a920d75f6138-kube-api-access-txh8f\") pod \"6db020e0-c316-47b9-ad20-a920d75f6138\" (UID: \"6db020e0-c316-47b9-ad20-a920d75f6138\") " Oct 31 20:06:00.348951 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.343773 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-home\" (UniqueName: \"kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-home\") pod \"6db020e0-c316-47b9-ad20-a920d75f6138\" (UID: \"6db020e0-c316-47b9-ad20-a920d75f6138\") " Oct 31 20:06:00.348951 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.343863 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-tmp-dir\") pod \"6db020e0-c316-47b9-ad20-a920d75f6138\" (UID: \"6db020e0-c316-47b9-ad20-a920d75f6138\") " Oct 31 20:06:00.348951 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.343897 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-model-cache\") pod \"6db020e0-c316-47b9-ad20-a920d75f6138\" (UID: \"6db020e0-c316-47b9-ad20-a920d75f6138\") " Oct 31 20:06:00.348951 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.344475 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-model-cache\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:00.348951 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.344498 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77506413-2985-4c17-b98c-dc414769ec5d-dshm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:00.348951 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.344514 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77506413-2985-4c17-b98c-dc414769ec5d-tls-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:00.348951 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.344528 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk5wh\" (UniqueName: \"kubernetes.io/projected/77506413-2985-4c17-b98c-dc414769ec5d-kube-api-access-wk5wh\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:00.348951 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.344773 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-model-cache" pod "6db020e0-c316-47b9-ad20-a920d75f6138" (UID: "6db020e0-c316-47b9-ad20-a920d75f6138"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:00.348951 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.345306 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-home" pod "6db020e0-c316-47b9-ad20-a920d75f6138" (UID: "6db020e0-c316-47b9-ad20-a920d75f6138"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:00.348951 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.345490 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-tmp-dir" pod "6db020e0-c316-47b9-ad20-a920d75f6138" (UID: "6db020e0-c316-47b9-ad20-a920d75f6138"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:00.348951 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.346340 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c173d8-4tkwr"] Oct 31 20:06:00.348951 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.348665 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-dshm" pod "6db020e0-c316-47b9-ad20-a920d75f6138" (UID: "6db020e0-c316-47b9-ad20-a920d75f6138"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:00.349942 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.349763 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db020e0-c316-47b9-ad20-a920d75f6138-kube-api-access-txh8f" pod "6db020e0-c316-47b9-ad20-a920d75f6138" (UID: "6db020e0-c316-47b9-ad20-a920d75f6138"). InnerVolumeSpecName "kube-api-access-txh8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:06:00.349942 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.349886 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db020e0-c316-47b9-ad20-a920d75f6138-tls-certs" pod "6db020e0-c316-47b9-ad20-a920d75f6138" (UID: "6db020e0-c316-47b9-ad20-a920d75f6138"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:06:00.458955 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.455156 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-dshm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:00.458955 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.455196 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db020e0-c316-47b9-ad20-a920d75f6138-tls-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:00.458955 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.455212 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txh8f\" (UniqueName: \"kubernetes.io/projected/6db020e0-c316-47b9-ad20-a920d75f6138-kube-api-access-txh8f\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:00.458955 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.455228 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-home\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:00.458955 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.455241 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-tmp-dir\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:00.458955 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.455253 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db020e0-c316-47b9-ad20-a920d75f6138-model-cache\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:00.552458 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.552414 4910 generic.go:356] "Generic (PLEG): container finished" podID="6db020e0-c316-47b9-ad20-a920d75f6138" containerID="c1eba0f875ec3982501b1251bc2c1fa1b6980d1ffe1cfb6e0b18def31e13e8de" exitCode=0 Oct 31 20:06:00.552667 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.552458 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" Oct 31 20:06:00.552667 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.552493 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" event={"ID":"6db020e0-c316-47b9-ad20-a920d75f6138","Type":"ContainerDied","Data":"c1eba0f875ec3982501b1251bc2c1fa1b6980d1ffe1cfb6e0b18def31e13e8de"} Oct 31 20:06:00.552667 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.552536 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7" event={"ID":"6db020e0-c316-47b9-ad20-a920d75f6138","Type":"ContainerDied","Data":"68ffe0ebacd6bbdaa05c32f94aec9f5a1d9a8497d384112309dfbe254ba71943"} Oct 31 20:06:00.552667 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.552556 4910 scope.go:122] "RemoveContainer" containerID="c1eba0f875ec3982501b1251bc2c1fa1b6980d1ffe1cfb6e0b18def31e13e8de" Oct 31 20:06:00.554290 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.554261 4910 generic.go:356] "Generic (PLEG): container finished" podID="77506413-2985-4c17-b98c-dc414769ec5d" containerID="6d3c3829dd2141679e7092f74dd2f4742aa00d1440e5d42dd81ead069e4465fe" exitCode=0 Oct 31 20:06:00.554522 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.554503 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" event={"ID":"77506413-2985-4c17-b98c-dc414769ec5d","Type":"ContainerDied","Data":"6d3c3829dd2141679e7092f74dd2f4742aa00d1440e5d42dd81ead069e4465fe"} Oct 31 20:06:00.554592 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.554539 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" event={"ID":"77506413-2985-4c17-b98c-dc414769ec5d","Type":"ContainerDied","Data":"5cb9ad3d63e083df8d1c1769f75ad8d614cb51c1591de318e3d4f1da95d2de8d"} Oct 31 20:06:00.554654 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.554596 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8" Oct 31 20:06:00.564479 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.564460 4910 scope.go:122] "RemoveContainer" containerID="c1eba0f875ec3982501b1251bc2c1fa1b6980d1ffe1cfb6e0b18def31e13e8de" Oct 31 20:06:00.564786 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:06:00.564761 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1eba0f875ec3982501b1251bc2c1fa1b6980d1ffe1cfb6e0b18def31e13e8de\": container with ID starting with c1eba0f875ec3982501b1251bc2c1fa1b6980d1ffe1cfb6e0b18def31e13e8de not found: ID does not exist" containerID="c1eba0f875ec3982501b1251bc2c1fa1b6980d1ffe1cfb6e0b18def31e13e8de" Oct 31 20:06:00.564854 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.564796 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1eba0f875ec3982501b1251bc2c1fa1b6980d1ffe1cfb6e0b18def31e13e8de"} err="failed to get container status \"c1eba0f875ec3982501b1251bc2c1fa1b6980d1ffe1cfb6e0b18def31e13e8de\": rpc error: code = NotFound desc = could not find container \"c1eba0f875ec3982501b1251bc2c1fa1b6980d1ffe1cfb6e0b18def31e13e8de\": container with ID starting with c1eba0f875ec3982501b1251bc2c1fa1b6980d1ffe1cfb6e0b18def31e13e8de not found: ID does not exist" Oct 31 20:06:00.564854 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.564812 4910 scope.go:122] "RemoveContainer" containerID="6d3c3829dd2141679e7092f74dd2f4742aa00d1440e5d42dd81ead069e4465fe" Oct 31 20:06:00.574163 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.574142 4910 scope.go:122] "RemoveContainer" containerID="6d3c3829dd2141679e7092f74dd2f4742aa00d1440e5d42dd81ead069e4465fe" Oct 31 20:06:00.574439 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:06:00.574421 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d3c3829dd2141679e7092f74dd2f4742aa00d1440e5d42dd81ead069e4465fe\": container with ID starting with 6d3c3829dd2141679e7092f74dd2f4742aa00d1440e5d42dd81ead069e4465fe not found: ID does not exist" containerID="6d3c3829dd2141679e7092f74dd2f4742aa00d1440e5d42dd81ead069e4465fe" Oct 31 20:06:00.574503 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.574444 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d3c3829dd2141679e7092f74dd2f4742aa00d1440e5d42dd81ead069e4465fe"} err="failed to get container status \"6d3c3829dd2141679e7092f74dd2f4742aa00d1440e5d42dd81ead069e4465fe\": rpc error: code = NotFound desc = could not find container \"6d3c3829dd2141679e7092f74dd2f4742aa00d1440e5d42dd81ead069e4465fe\": container with ID starting with 6d3c3829dd2141679e7092f74dd2f4742aa00d1440e5d42dd81ead069e4465fe not found: ID does not exist" Oct 31 20:06:00.588271 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.588250 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7"] Oct 31 20:06:00.591381 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.591359 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-8730d5/simulator-w6-8730d5-kserve-5754955c6b-5vxr7"] Oct 31 20:06:00.607460 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.607437 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8"] Oct 31 20:06:00.617435 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:00.617413 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-8730d5/distinct-2-w6-8730d5-kserve-f8ff54685-cswm8"] Oct 31 20:06:01.561702 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.561663 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6668e56-vzzz8"] Oct 31 20:06:01.562461 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.562440 4910 cpu_manager.go:414] "containerMap: removing container" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" Oct 31 20:06:01.562563 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.562463 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" Oct 31 20:06:01.562563 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.562486 4910 cpu_manager.go:414] "containerMap: removing container" podUID="204c09e1-cfce-4add-986d-165e78767839" containerName="istio-proxy" Oct 31 20:06:01.562563 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.562495 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204c09e1-cfce-4add-986d-165e78767839" containerName="istio-proxy" Oct 31 20:06:01.562563 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.562515 4910 cpu_manager.go:414] "containerMap: removing container" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" Oct 31 20:06:01.562563 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.562523 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" Oct 31 20:06:01.562563 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.562541 4910 cpu_manager.go:414] "containerMap: removing container" podUID="204c09e1-cfce-4add-986d-165e78767839" containerName="istio-proxy" Oct 31 20:06:01.562563 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.562548 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204c09e1-cfce-4add-986d-165e78767839" containerName="istio-proxy" Oct 31 20:06:01.562899 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.562604 4910 cpu_manager.go:414] "containerMap: removing container" podUID="901b6b03-183b-4819-a239-7ae5565e9eba" containerName="revoke-keys" Oct 31 20:06:01.562899 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.562613 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="901b6b03-183b-4819-a239-7ae5565e9eba" containerName="revoke-keys" Oct 31 20:06:01.562899 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.562726 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="949b61a5-f07e-47c1-b360-a3bfde012f42" containerName="istio-proxy" Oct 31 20:06:01.562899 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.562743 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="901b6b03-183b-4819-a239-7ae5565e9eba" containerName="revoke-keys" Oct 31 20:06:01.562899 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.562753 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="77506413-2985-4c17-b98c-dc414769ec5d" containerName="main" Oct 31 20:06:01.562899 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.562760 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="204c09e1-cfce-4add-986d-165e78767839" containerName="istio-proxy" Oct 31 20:06:01.562899 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.562770 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" containerName="main" Oct 31 20:06:01.568145 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.568124 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6668e56-vzzz8" Oct 31 20:06:01.572480 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.572451 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wtmr\" (UniqueName: \"kubernetes.io/projected/0e75dc23-b035-493b-9528-f24b18b140bd-kube-api-access-7wtmr\") pod \"maas-api-revoke-sub-d6668e56-vzzz8\" (UID: \"0e75dc23-b035-493b-9528-f24b18b140bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6668e56-vzzz8" Oct 31 20:06:01.572615 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.572572 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e75dc23-b035-493b-9528-f24b18b140bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6668e56-vzzz8\" (UID: \"0e75dc23-b035-493b-9528-f24b18b140bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6668e56-vzzz8" Oct 31 20:06:01.572668 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.572613 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e75dc23-b035-493b-9528-f24b18b140bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6668e56-vzzz8\" (UID: \"0e75dc23-b035-493b-9528-f24b18b140bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6668e56-vzzz8" Oct 31 20:06:01.586020 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.585992 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6668e56-vzzz8"] Oct 31 20:06:01.586877 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.586824 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" containerID="cri-o://4bbdbd8807904a3d2b84a0c7644ee280c605a5794e74010f046eaeba69c58757" gracePeriod=45 Oct 31 20:06:01.617947 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.596794 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" containerID="cri-o://a8dca1b3e4cfbee24c6eafc0525b5ce3ca7cc2245527ea1bda7c9c4da4a06418" gracePeriod=45 Oct 31 20:06:01.617947 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.617388 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:06:01.617947 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.617425 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 20:06:01.673379 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.673339 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wtmr\" (UniqueName: \"kubernetes.io/projected/0e75dc23-b035-493b-9528-f24b18b140bd-kube-api-access-7wtmr\") pod \"maas-api-revoke-sub-d6668e56-vzzz8\" (UID: \"0e75dc23-b035-493b-9528-f24b18b140bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6668e56-vzzz8" Oct 31 20:06:01.673565 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.673396 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e75dc23-b035-493b-9528-f24b18b140bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6668e56-vzzz8\" (UID: \"0e75dc23-b035-493b-9528-f24b18b140bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6668e56-vzzz8" Oct 31 20:06:01.673565 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.673417 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e75dc23-b035-493b-9528-f24b18b140bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6668e56-vzzz8\" (UID: \"0e75dc23-b035-493b-9528-f24b18b140bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6668e56-vzzz8" Oct 31 20:06:01.674090 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.674060 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e75dc23-b035-493b-9528-f24b18b140bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6668e56-vzzz8\" (UID: \"0e75dc23-b035-493b-9528-f24b18b140bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6668e56-vzzz8" Oct 31 20:06:01.675775 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.675749 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e75dc23-b035-493b-9528-f24b18b140bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6668e56-vzzz8\" (UID: \"0e75dc23-b035-493b-9528-f24b18b140bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6668e56-vzzz8" Oct 31 20:06:01.687337 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.687309 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wtmr\" (UniqueName: \"kubernetes.io/projected/0e75dc23-b035-493b-9528-f24b18b140bd-kube-api-access-7wtmr\") pod \"maas-api-revoke-sub-d6668e56-vzzz8\" (UID: \"0e75dc23-b035-493b-9528-f24b18b140bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6668e56-vzzz8" Oct 31 20:06:01.759060 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.759020 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db020e0-c316-47b9-ad20-a920d75f6138" path="/var/lib/kubelet/pods/6db020e0-c316-47b9-ad20-a920d75f6138/volumes" Oct 31 20:06:01.759681 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.759661 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77506413-2985-4c17-b98c-dc414769ec5d" path="/var/lib/kubelet/pods/77506413-2985-4c17-b98c-dc414769ec5d/volumes" Oct 31 20:06:01.761792 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.760255 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79" path="/var/lib/kubelet/pods/aa6e068f-8ca6-4ce8-91ce-e5d324c5bf79/volumes" Oct 31 20:06:01.895559 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.891177 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6668e56-vzzz8" Oct 31 20:06:01.957248 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.957154 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 20:06:01.961090 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.961061 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 20:06:01.981364 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.981324 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/902b298d-646c-45e4-bff8-d505de51c689-tls-certs\" (UniqueName: \"kubernetes.io/secret/902b298d-646c-45e4-bff8-d505de51c689-tls-certs\") pod \"902b298d-646c-45e4-bff8-d505de51c689\" (UID: \"902b298d-646c-45e4-bff8-d505de51c689\") " Oct 31 20:06:01.981534 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.981394 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3803d62c-0b0f-49ce-a755-0db6506489d4-kube-api-access-l9jfn\" (UniqueName: \"kubernetes.io/projected/3803d62c-0b0f-49ce-a755-0db6506489d4-kube-api-access-l9jfn\") pod \"3803d62c-0b0f-49ce-a755-0db6506489d4\" (UID: \"3803d62c-0b0f-49ce-a755-0db6506489d4\") " Oct 31 20:06:01.981534 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.981437 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-model-cache\") pod \"3803d62c-0b0f-49ce-a755-0db6506489d4\" (UID: \"3803d62c-0b0f-49ce-a755-0db6506489d4\") " Oct 31 20:06:01.981534 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.981480 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-model-cache\") pod \"902b298d-646c-45e4-bff8-d505de51c689\" (UID: \"902b298d-646c-45e4-bff8-d505de51c689\") " Oct 31 20:06:01.981642 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.981549 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-tmp-dir\") pod \"3803d62c-0b0f-49ce-a755-0db6506489d4\" (UID: \"3803d62c-0b0f-49ce-a755-0db6506489d4\") " Oct 31 20:06:01.981642 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.981625 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-home\" (UniqueName: \"kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-home\") pod \"3803d62c-0b0f-49ce-a755-0db6506489d4\" (UID: \"3803d62c-0b0f-49ce-a755-0db6506489d4\") " Oct 31 20:06:01.981711 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.981660 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-dshm\" (UniqueName: \"kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-dshm\") pod \"902b298d-646c-45e4-bff8-d505de51c689\" (UID: \"902b298d-646c-45e4-bff8-d505de51c689\") " Oct 31 20:06:01.981711 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.981686 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-home\" (UniqueName: \"kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-home\") pod \"902b298d-646c-45e4-bff8-d505de51c689\" (UID: \"902b298d-646c-45e4-bff8-d505de51c689\") " Oct 31 20:06:01.981772 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.981713 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3803d62c-0b0f-49ce-a755-0db6506489d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/3803d62c-0b0f-49ce-a755-0db6506489d4-tls-certs\") pod \"3803d62c-0b0f-49ce-a755-0db6506489d4\" (UID: \"3803d62c-0b0f-49ce-a755-0db6506489d4\") " Oct 31 20:06:01.981808 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.981782 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/902b298d-646c-45e4-bff8-d505de51c689-kube-api-access-slph9\" (UniqueName: \"kubernetes.io/projected/902b298d-646c-45e4-bff8-d505de51c689-kube-api-access-slph9\") pod \"902b298d-646c-45e4-bff8-d505de51c689\" (UID: \"902b298d-646c-45e4-bff8-d505de51c689\") " Oct 31 20:06:01.981844 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.981807 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-tmp-dir\") pod \"902b298d-646c-45e4-bff8-d505de51c689\" (UID: \"902b298d-646c-45e4-bff8-d505de51c689\") " Oct 31 20:06:01.981969 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.981951 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-dshm\") pod \"3803d62c-0b0f-49ce-a755-0db6506489d4\" (UID: \"3803d62c-0b0f-49ce-a755-0db6506489d4\") " Oct 31 20:06:01.983118 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.983073 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-home" pod "3803d62c-0b0f-49ce-a755-0db6506489d4" (UID: "3803d62c-0b0f-49ce-a755-0db6506489d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:01.983605 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.983582 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-model-cache" pod "3803d62c-0b0f-49ce-a755-0db6506489d4" (UID: "3803d62c-0b0f-49ce-a755-0db6506489d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:01.993773 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.985326 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-dshm" pod "3803d62c-0b0f-49ce-a755-0db6506489d4" (UID: "3803d62c-0b0f-49ce-a755-0db6506489d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:01.993773 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.985565 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-tmp-dir" pod "902b298d-646c-45e4-bff8-d505de51c689" (UID: "902b298d-646c-45e4-bff8-d505de51c689"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:01.997287 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.997086 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3803d62c-0b0f-49ce-a755-0db6506489d4-tls-certs" pod "3803d62c-0b0f-49ce-a755-0db6506489d4" (UID: "3803d62c-0b0f-49ce-a755-0db6506489d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:06:01.997660 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.997468 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-home" pod "902b298d-646c-45e4-bff8-d505de51c689" (UID: "902b298d-646c-45e4-bff8-d505de51c689"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:01.997660 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.997570 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3803d62c-0b0f-49ce-a755-0db6506489d4-kube-api-access-l9jfn" pod "3803d62c-0b0f-49ce-a755-0db6506489d4" (UID: "3803d62c-0b0f-49ce-a755-0db6506489d4"). InnerVolumeSpecName "kube-api-access-l9jfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:06:01.997824 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.997787 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-tmp-dir" pod "3803d62c-0b0f-49ce-a755-0db6506489d4" (UID: "3803d62c-0b0f-49ce-a755-0db6506489d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:01.997995 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:01.997964 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-model-cache" pod "902b298d-646c-45e4-bff8-d505de51c689" (UID: "902b298d-646c-45e4-bff8-d505de51c689"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:02.004074 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.004036 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902b298d-646c-45e4-bff8-d505de51c689-tls-certs" pod "902b298d-646c-45e4-bff8-d505de51c689" (UID: "902b298d-646c-45e4-bff8-d505de51c689"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:06:02.004185 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.004053 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902b298d-646c-45e4-bff8-d505de51c689-kube-api-access-slph9" pod "902b298d-646c-45e4-bff8-d505de51c689" (UID: "902b298d-646c-45e4-bff8-d505de51c689"). InnerVolumeSpecName "kube-api-access-slph9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:06:02.006420 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.006376 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-dshm" pod "902b298d-646c-45e4-bff8-d505de51c689" (UID: "902b298d-646c-45e4-bff8-d505de51c689"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:02.083824 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.083738 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slph9\" (UniqueName: \"kubernetes.io/projected/902b298d-646c-45e4-bff8-d505de51c689-kube-api-access-slph9\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:02.083824 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.083771 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-tmp-dir\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:02.083824 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.083783 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-dshm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:02.083824 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.083791 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902b298d-646c-45e4-bff8-d505de51c689-tls-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:02.083824 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.083799 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9jfn\" (UniqueName: \"kubernetes.io/projected/3803d62c-0b0f-49ce-a755-0db6506489d4-kube-api-access-l9jfn\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:02.083824 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.083810 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-model-cache\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:02.083824 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.083819 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-model-cache\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:02.083824 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.083826 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-tmp-dir\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:02.084426 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.083835 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3803d62c-0b0f-49ce-a755-0db6506489d4-home\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:02.084426 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.083843 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-dshm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:02.084426 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.083849 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902b298d-646c-45e4-bff8-d505de51c689-home\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:02.084426 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.083857 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3803d62c-0b0f-49ce-a755-0db6506489d4-tls-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:02.251741 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.251710 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6668e56-vzzz8"] Oct 31 20:06:02.252053 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:06:02.252030 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e75dc23_b035_493b_9528_f24b18b140bd.slice/crio-27dcc992675f826439cd253fcf248582680e16824f522fd84689a095a26f1f90 WatchSource:0}: Error finding container 27dcc992675f826439cd253fcf248582680e16824f522fd84689a095a26f1f90: Status 404 returned error can't find the container with id 27dcc992675f826439cd253fcf248582680e16824f522fd84689a095a26f1f90 Oct 31 20:06:02.574929 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.574870 4910 generic.go:356] "Generic (PLEG): container finished" podID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerID="4bbdbd8807904a3d2b84a0c7644ee280c605a5794e74010f046eaeba69c58757" exitCode=0 Oct 31 20:06:02.575425 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.574940 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" Oct 31 20:06:02.575425 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.574962 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" event={"ID":"3803d62c-0b0f-49ce-a755-0db6506489d4","Type":"ContainerDied","Data":"4bbdbd8807904a3d2b84a0c7644ee280c605a5794e74010f046eaeba69c58757"} Oct 31 20:06:02.575425 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.575008 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8" event={"ID":"3803d62c-0b0f-49ce-a755-0db6506489d4","Type":"ContainerDied","Data":"c2ecc0ee2b5b33313c1a0580245722c55e52058f75781fb1e9affe14d0204544"} Oct 31 20:06:02.575425 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.575027 4910 scope.go:122] "RemoveContainer" containerID="4bbdbd8807904a3d2b84a0c7644ee280c605a5794e74010f046eaeba69c58757" Oct 31 20:06:02.576786 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.576757 4910 generic.go:356] "Generic (PLEG): container finished" podID="902b298d-646c-45e4-bff8-d505de51c689" containerID="a8dca1b3e4cfbee24c6eafc0525b5ce3ca7cc2245527ea1bda7c9c4da4a06418" exitCode=0 Oct 31 20:06:02.576938 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.576789 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" Oct 31 20:06:02.576938 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.576854 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" event={"ID":"902b298d-646c-45e4-bff8-d505de51c689","Type":"ContainerDied","Data":"a8dca1b3e4cfbee24c6eafc0525b5ce3ca7cc2245527ea1bda7c9c4da4a06418"} Oct 31 20:06:02.576938 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.576896 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz" event={"ID":"902b298d-646c-45e4-bff8-d505de51c689","Type":"ContainerDied","Data":"c8b567521b56f9d30c61dd973f656359961d5d7e3ed251f5bb89cf43507cc848"} Oct 31 20:06:02.578882 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.578856 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6668e56-vzzz8" event={"ID":"0e75dc23-b035-493b-9528-f24b18b140bd","Type":"ContainerStarted","Data":"a7f0291d1d6ade3d5cc0a813ff9502f9da70ffee0981dba5141ab0e92bd7ca07"} Oct 31 20:06:02.579015 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.578941 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6668e56-vzzz8" event={"ID":"0e75dc23-b035-493b-9528-f24b18b140bd","Type":"ContainerStarted","Data":"27dcc992675f826439cd253fcf248582680e16824f522fd84689a095a26f1f90"} Oct 31 20:06:02.586378 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.586295 4910 scope.go:122] "RemoveContainer" containerID="4bbdbd8807904a3d2b84a0c7644ee280c605a5794e74010f046eaeba69c58757" Oct 31 20:06:02.586800 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:06:02.586753 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bbdbd8807904a3d2b84a0c7644ee280c605a5794e74010f046eaeba69c58757\": container with ID starting with 4bbdbd8807904a3d2b84a0c7644ee280c605a5794e74010f046eaeba69c58757 not found: ID does not exist" containerID="4bbdbd8807904a3d2b84a0c7644ee280c605a5794e74010f046eaeba69c58757" Oct 31 20:06:02.586800 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.586785 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bbdbd8807904a3d2b84a0c7644ee280c605a5794e74010f046eaeba69c58757"} err="failed to get container status \"4bbdbd8807904a3d2b84a0c7644ee280c605a5794e74010f046eaeba69c58757\": rpc error: code = NotFound desc = could not find container \"4bbdbd8807904a3d2b84a0c7644ee280c605a5794e74010f046eaeba69c58757\": container with ID starting with 4bbdbd8807904a3d2b84a0c7644ee280c605a5794e74010f046eaeba69c58757 not found: ID does not exist" Oct 31 20:06:02.586995 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.586807 4910 scope.go:122] "RemoveContainer" containerID="a8dca1b3e4cfbee24c6eafc0525b5ce3ca7cc2245527ea1bda7c9c4da4a06418" Oct 31 20:06:02.596496 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.596476 4910 scope.go:122] "RemoveContainer" containerID="a8dca1b3e4cfbee24c6eafc0525b5ce3ca7cc2245527ea1bda7c9c4da4a06418" Oct 31 20:06:02.596745 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:06:02.596726 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8dca1b3e4cfbee24c6eafc0525b5ce3ca7cc2245527ea1bda7c9c4da4a06418\": container with ID starting with a8dca1b3e4cfbee24c6eafc0525b5ce3ca7cc2245527ea1bda7c9c4da4a06418 not found: ID does not exist" containerID="a8dca1b3e4cfbee24c6eafc0525b5ce3ca7cc2245527ea1bda7c9c4da4a06418" Oct 31 20:06:02.596812 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.596752 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8dca1b3e4cfbee24c6eafc0525b5ce3ca7cc2245527ea1bda7c9c4da4a06418"} err="failed to get container status \"a8dca1b3e4cfbee24c6eafc0525b5ce3ca7cc2245527ea1bda7c9c4da4a06418\": rpc error: code = NotFound desc = could not find container \"a8dca1b3e4cfbee24c6eafc0525b5ce3ca7cc2245527ea1bda7c9c4da4a06418\": container with ID starting with a8dca1b3e4cfbee24c6eafc0525b5ce3ca7cc2245527ea1bda7c9c4da4a06418 not found: ID does not exist" Oct 31 20:06:02.613399 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.613342 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6668e56-vzzz8" podStartSLOduration=1.613326485 podStartE2EDuration="1.613326485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:06:02.611785219 +0000 UTC m=+2796.974605592" watchObservedRunningTime="2026-10-31 20:06:02.613326485 +0000 UTC m=+2796.976146858" Oct 31 20:06:02.645023 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.644988 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz"] Oct 31 20:06:02.650662 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.650633 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-aeb292/premium-w2-aeb292-kserve-b5975b895-7pglz"] Oct 31 20:06:02.686012 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.685971 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8"] Oct 31 20:06:02.689151 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:02.689127 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-aeb292/distinct-w2-aeb292-kserve-5c7b9b66b6-k5hg8"] Oct 31 20:06:03.394793 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:03.394732 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9" podUID="c34f06bd-3b50-4342-9719-bcccd4be955f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:06:03.592260 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:03.592217 4910 generic.go:356] "Generic (PLEG): container finished" podID="0e75dc23-b035-493b-9528-f24b18b140bd" containerID="a7f0291d1d6ade3d5cc0a813ff9502f9da70ffee0981dba5141ab0e92bd7ca07" exitCode=0 Oct 31 20:06:03.592783 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:03.592264 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6668e56-vzzz8" event={"ID":"0e75dc23-b035-493b-9528-f24b18b140bd","Type":"ContainerDied","Data":"a7f0291d1d6ade3d5cc0a813ff9502f9da70ffee0981dba5141ab0e92bd7ca07"} Oct 31 20:06:03.755139 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:03.755104 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" path="/var/lib/kubelet/pods/3803d62c-0b0f-49ce-a755-0db6506489d4/volumes" Oct 31 20:06:03.755630 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:03.755615 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="902b298d-646c-45e4-bff8-d505de51c689" path="/var/lib/kubelet/pods/902b298d-646c-45e4-bff8-d505de51c689/volumes" Oct 31 20:06:04.409803 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:04.409762 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq" podUID="7d56bb36-ed0e-4127-9461-cbec40a46b95" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:06:04.450351 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:04.450313 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn"] Oct 31 20:06:04.458760 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:04.458725 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aeb292-vwfdn"] Oct 31 20:06:04.767527 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:04.767500 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6668e56-vzzz8" Oct 31 20:06:04.820940 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:04.820892 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e75dc23-b035-493b-9528-f24b18b140bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e75dc23-b035-493b-9528-f24b18b140bd-maas-api-xks-ca\") pod \"0e75dc23-b035-493b-9528-f24b18b140bd\" (UID: \"0e75dc23-b035-493b-9528-f24b18b140bd\") " Oct 31 20:06:04.821135 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:04.821027 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e75dc23-b035-493b-9528-f24b18b140bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e75dc23-b035-493b-9528-f24b18b140bd-maas-api-openshift-ca\") pod \"0e75dc23-b035-493b-9528-f24b18b140bd\" (UID: \"0e75dc23-b035-493b-9528-f24b18b140bd\") " Oct 31 20:06:04.821135 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:04.821106 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e75dc23-b035-493b-9528-f24b18b140bd-kube-api-access-7wtmr\" (UniqueName: \"kubernetes.io/projected/0e75dc23-b035-493b-9528-f24b18b140bd-kube-api-access-7wtmr\") pod \"0e75dc23-b035-493b-9528-f24b18b140bd\" (UID: \"0e75dc23-b035-493b-9528-f24b18b140bd\") " Oct 31 20:06:04.822014 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:04.821704 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e75dc23-b035-493b-9528-f24b18b140bd-maas-api-openshift-ca" pod "0e75dc23-b035-493b-9528-f24b18b140bd" (UID: "0e75dc23-b035-493b-9528-f24b18b140bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:06:04.823800 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:04.823764 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e75dc23-b035-493b-9528-f24b18b140bd-maas-api-xks-ca" pod "0e75dc23-b035-493b-9528-f24b18b140bd" (UID: "0e75dc23-b035-493b-9528-f24b18b140bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:06:04.824105 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:04.824081 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e75dc23-b035-493b-9528-f24b18b140bd-kube-api-access-7wtmr" pod "0e75dc23-b035-493b-9528-f24b18b140bd" (UID: "0e75dc23-b035-493b-9528-f24b18b140bd"). InnerVolumeSpecName "kube-api-access-7wtmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:06:04.922595 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:04.922562 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e75dc23-b035-493b-9528-f24b18b140bd-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:04.922595 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:04.922597 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wtmr\" (UniqueName: \"kubernetes.io/projected/0e75dc23-b035-493b-9528-f24b18b140bd-kube-api-access-7wtmr\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:04.922778 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:04.922616 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e75dc23-b035-493b-9528-f24b18b140bd-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:05.637852 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:05.637819 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6668e56-vzzz8" Oct 31 20:06:05.637852 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:05.637830 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6668e56-vzzz8" event={"ID":"0e75dc23-b035-493b-9528-f24b18b140bd","Type":"ContainerDied","Data":"27dcc992675f826439cd253fcf248582680e16824f522fd84689a095a26f1f90"} Oct 31 20:06:05.638119 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:05.637892 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27dcc992675f826439cd253fcf248582680e16824f522fd84689a095a26f1f90" Oct 31 20:06:05.754743 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:05.754709 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a673180d-8ebc-474d-a6d1-130ea7603aed" path="/var/lib/kubelet/pods/a673180d-8ebc-474d-a6d1-130ea7603aed/volumes" Oct 31 20:06:12.659476 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:12.659433 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk"] Oct 31 20:06:12.664333 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:12.664274 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" podUID="ce5603fa-39d7-4705-a0e3-85297ad53a78" containerName="istio-proxy" containerID="cri-o://a28a4b487bc10e6a168d3b4871bfc85bd6870d6d79658c96328a29b279fb44a1" gracePeriod=30 Oct 31 20:06:12.724433 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:06:12.724388 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:06:13.052007 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:13.051905 4910 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:06:13.052007 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:13.051986 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" podUID="ce5603fa-39d7-4705-a0e3-85297ad53a78" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:06:13.400932 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:13.400661 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9" podUID="c34f06bd-3b50-4342-9719-bcccd4be955f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:06:14.408981 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:14.408939 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq" podUID="7d56bb36-ed0e-4127-9461-cbec40a46b95" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:06:15.364379 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:15.364317 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq" Oct 31 20:06:15.491967 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:15.491906 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d56bb36-ed0e-4127-9461-cbec40a46b95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d56bb36-ed0e-4127-9461-cbec40a46b95-plugins-config-volume\") pod \"7d56bb36-ed0e-4127-9461-cbec40a46b95\" (UID: \"7d56bb36-ed0e-4127-9461-cbec40a46b95\") " Oct 31 20:06:15.492454 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:15.492060 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d56bb36-ed0e-4127-9461-cbec40a46b95-kube-api-access-mj5w6\" (UniqueName: \"kubernetes.io/projected/7d56bb36-ed0e-4127-9461-cbec40a46b95-kube-api-access-mj5w6\") pod \"7d56bb36-ed0e-4127-9461-cbec40a46b95\" (UID: \"7d56bb36-ed0e-4127-9461-cbec40a46b95\") " Oct 31 20:06:15.492454 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:15.492277 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d56bb36-ed0e-4127-9461-cbec40a46b95-plugins-config-volume" pod "7d56bb36-ed0e-4127-9461-cbec40a46b95" (UID: "7d56bb36-ed0e-4127-9461-cbec40a46b95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:06:15.492454 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:15.492385 4910 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d56bb36-ed0e-4127-9461-cbec40a46b95-plugins-config-volume\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:15.494209 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:15.494179 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d56bb36-ed0e-4127-9461-cbec40a46b95-kube-api-access-mj5w6" pod "7d56bb36-ed0e-4127-9461-cbec40a46b95" (UID: "7d56bb36-ed0e-4127-9461-cbec40a46b95"). InnerVolumeSpecName "kube-api-access-mj5w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:06:15.594001 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:15.593967 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj5w6\" (UniqueName: \"kubernetes.io/projected/7d56bb36-ed0e-4127-9461-cbec40a46b95-kube-api-access-mj5w6\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:15.833082 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:15.833042 4910 generic.go:356] "Generic (PLEG): container finished" podID="7d56bb36-ed0e-4127-9461-cbec40a46b95" containerID="b9e5a644cfb960de024dd921569b10f28f4c48cc6a8c226291b4a417e6ee3b35" exitCode=0 Oct 31 20:06:15.833284 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:15.833090 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq" Oct 31 20:06:15.833284 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:15.833126 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq" event={"ID":"7d56bb36-ed0e-4127-9461-cbec40a46b95","Type":"ContainerDied","Data":"b9e5a644cfb960de024dd921569b10f28f4c48cc6a8c226291b4a417e6ee3b35"} Oct 31 20:06:15.833284 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:15.833194 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq" event={"ID":"7d56bb36-ed0e-4127-9461-cbec40a46b95","Type":"ContainerDied","Data":"74df750d778ee9dc550afdf9e601fc8cc5a36668c0afbbaa38898ec35c14b3cd"} Oct 31 20:06:15.833284 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:15.833225 4910 scope.go:122] "RemoveContainer" containerID="b9e5a644cfb960de024dd921569b10f28f4c48cc6a8c226291b4a417e6ee3b35" Oct 31 20:06:15.843744 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:15.843722 4910 scope.go:122] "RemoveContainer" containerID="b9e5a644cfb960de024dd921569b10f28f4c48cc6a8c226291b4a417e6ee3b35" Oct 31 20:06:15.844080 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:06:15.844052 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9e5a644cfb960de024dd921569b10f28f4c48cc6a8c226291b4a417e6ee3b35\": container with ID starting with b9e5a644cfb960de024dd921569b10f28f4c48cc6a8c226291b4a417e6ee3b35 not found: ID does not exist" containerID="b9e5a644cfb960de024dd921569b10f28f4c48cc6a8c226291b4a417e6ee3b35" Oct 31 20:06:15.844144 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:15.844088 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9e5a644cfb960de024dd921569b10f28f4c48cc6a8c226291b4a417e6ee3b35"} err="failed to get container status \"b9e5a644cfb960de024dd921569b10f28f4c48cc6a8c226291b4a417e6ee3b35\": rpc error: code = NotFound desc = could not find container \"b9e5a644cfb960de024dd921569b10f28f4c48cc6a8c226291b4a417e6ee3b35\": container with ID starting with b9e5a644cfb960de024dd921569b10f28f4c48cc6a8c226291b4a417e6ee3b35 not found: ID does not exist" Oct 31 20:06:15.861025 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:15.860997 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq"] Oct 31 20:06:15.868363 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:15.868341 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-8730d5-7f459f7c9c-ss9sq"] Oct 31 20:06:17.174775 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:17.174718 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9" Oct 31 20:06:17.325051 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:17.325013 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34f06bd-3b50-4342-9719-bcccd4be955f-kube-api-access-gfx9l\" (UniqueName: \"kubernetes.io/projected/c34f06bd-3b50-4342-9719-bcccd4be955f-kube-api-access-gfx9l\") pod \"c34f06bd-3b50-4342-9719-bcccd4be955f\" (UID: \"c34f06bd-3b50-4342-9719-bcccd4be955f\") " Oct 31 20:06:17.325262 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:17.325081 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c34f06bd-3b50-4342-9719-bcccd4be955f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c34f06bd-3b50-4342-9719-bcccd4be955f-plugins-config-volume\") pod \"c34f06bd-3b50-4342-9719-bcccd4be955f\" (UID: \"c34f06bd-3b50-4342-9719-bcccd4be955f\") " Oct 31 20:06:17.325542 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:17.325511 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c34f06bd-3b50-4342-9719-bcccd4be955f-plugins-config-volume" pod "c34f06bd-3b50-4342-9719-bcccd4be955f" (UID: "c34f06bd-3b50-4342-9719-bcccd4be955f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:06:17.327258 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:17.327238 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34f06bd-3b50-4342-9719-bcccd4be955f-kube-api-access-gfx9l" pod "c34f06bd-3b50-4342-9719-bcccd4be955f" (UID: "c34f06bd-3b50-4342-9719-bcccd4be955f"). InnerVolumeSpecName "kube-api-access-gfx9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:06:17.426745 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:17.426649 4910 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c34f06bd-3b50-4342-9719-bcccd4be955f-plugins-config-volume\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:17.426745 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:17.426681 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfx9l\" (UniqueName: \"kubernetes.io/projected/c34f06bd-3b50-4342-9719-bcccd4be955f-kube-api-access-gfx9l\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:17.757254 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:17.757217 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d56bb36-ed0e-4127-9461-cbec40a46b95" path="/var/lib/kubelet/pods/7d56bb36-ed0e-4127-9461-cbec40a46b95/volumes" Oct 31 20:06:17.858467 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:17.858429 4910 generic.go:356] "Generic (PLEG): container finished" podID="c34f06bd-3b50-4342-9719-bcccd4be955f" containerID="938d28e68583a4153252ecef43fbb6e4c1e53561bcb89640f939bc7a91e6d39c" exitCode=0 Oct 31 20:06:17.858778 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:17.858470 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9" event={"ID":"c34f06bd-3b50-4342-9719-bcccd4be955f","Type":"ContainerDied","Data":"938d28e68583a4153252ecef43fbb6e4c1e53561bcb89640f939bc7a91e6d39c"} Oct 31 20:06:17.859057 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:17.858510 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9" Oct 31 20:06:17.859177 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:17.859043 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9" event={"ID":"c34f06bd-3b50-4342-9719-bcccd4be955f","Type":"ContainerDied","Data":"a26f04ac563f74403d5ad651d60849363345081672a2c68bf893b9197f292350"} Oct 31 20:06:17.859289 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:17.859276 4910 scope.go:122] "RemoveContainer" containerID="938d28e68583a4153252ecef43fbb6e4c1e53561bcb89640f939bc7a91e6d39c" Oct 31 20:06:17.861836 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:17.861685 4910 generic.go:356] "Generic (PLEG): container finished" podID="ce5603fa-39d7-4705-a0e3-85297ad53a78" containerID="a28a4b487bc10e6a168d3b4871bfc85bd6870d6d79658c96328a29b279fb44a1" exitCode=0 Oct 31 20:06:17.861836 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:17.861789 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" event={"ID":"ce5603fa-39d7-4705-a0e3-85297ad53a78","Type":"ContainerDied","Data":"a28a4b487bc10e6a168d3b4871bfc85bd6870d6d79658c96328a29b279fb44a1"} Oct 31 20:06:17.872477 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:17.872451 4910 scope.go:122] "RemoveContainer" containerID="938d28e68583a4153252ecef43fbb6e4c1e53561bcb89640f939bc7a91e6d39c" Oct 31 20:06:17.872845 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:06:17.872819 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"938d28e68583a4153252ecef43fbb6e4c1e53561bcb89640f939bc7a91e6d39c\": container with ID starting with 938d28e68583a4153252ecef43fbb6e4c1e53561bcb89640f939bc7a91e6d39c not found: ID does not exist" containerID="938d28e68583a4153252ecef43fbb6e4c1e53561bcb89640f939bc7a91e6d39c" Oct 31 20:06:17.873048 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:17.872855 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"938d28e68583a4153252ecef43fbb6e4c1e53561bcb89640f939bc7a91e6d39c"} err="failed to get container status \"938d28e68583a4153252ecef43fbb6e4c1e53561bcb89640f939bc7a91e6d39c\": rpc error: code = NotFound desc = could not find container \"938d28e68583a4153252ecef43fbb6e4c1e53561bcb89640f939bc7a91e6d39c\": container with ID starting with 938d28e68583a4153252ecef43fbb6e4c1e53561bcb89640f939bc7a91e6d39c not found: ID does not exist" Oct 31 20:06:17.884947 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:17.884906 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9"] Oct 31 20:06:17.889507 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:17.889479 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-aeb292-7b88bf8765-wcrb9"] Oct 31 20:06:17.997581 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:17.997540 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 20:06:18.147692 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.147646 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-data\") pod \"ce5603fa-39d7-4705-a0e3-85297ad53a78\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " Oct 31 20:06:18.147896 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.147719 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-workload-socket\") pod \"ce5603fa-39d7-4705-a0e3-85297ad53a78\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " Oct 31 20:06:18.147896 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.147760 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-envoy\") pod \"ce5603fa-39d7-4705-a0e3-85297ad53a78\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " Oct 31 20:06:18.147896 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.147831 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-credential-socket\") pod \"ce5603fa-39d7-4705-a0e3-85297ad53a78\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " Oct 31 20:06:18.148103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.147946 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-workload-certs\") pod \"ce5603fa-39d7-4705-a0e3-85297ad53a78\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " Oct 31 20:06:18.148103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.148006 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-podinfo\") pod \"ce5603fa-39d7-4705-a0e3-85297ad53a78\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " Oct 31 20:06:18.148103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.148018 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-data" pod "ce5603fa-39d7-4705-a0e3-85297ad53a78" (UID: "ce5603fa-39d7-4705-a0e3-85297ad53a78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:18.148103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.148040 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-workload-socket" pod "ce5603fa-39d7-4705-a0e3-85297ad53a78" (UID: "ce5603fa-39d7-4705-a0e3-85297ad53a78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:18.148316 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.148125 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce5603fa-39d7-4705-a0e3-85297ad53a78-kube-api-access-797c6\" (UniqueName: \"kubernetes.io/projected/ce5603fa-39d7-4705-a0e3-85297ad53a78-kube-api-access-797c6\") pod \"ce5603fa-39d7-4705-a0e3-85297ad53a78\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " Oct 31 20:06:18.148316 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.148164 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce5603fa-39d7-4705-a0e3-85297ad53a78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce5603fa-39d7-4705-a0e3-85297ad53a78-istiod-ca-cert\") pod \"ce5603fa-39d7-4705-a0e3-85297ad53a78\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " Oct 31 20:06:18.148316 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.148235 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-token\" (UniqueName: \"kubernetes.io/projected/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-token\") pod \"ce5603fa-39d7-4705-a0e3-85297ad53a78\" (UID: \"ce5603fa-39d7-4705-a0e3-85297ad53a78\") " Oct 31 20:06:18.148316 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.148239 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-credential-socket" pod "ce5603fa-39d7-4705-a0e3-85297ad53a78" (UID: "ce5603fa-39d7-4705-a0e3-85297ad53a78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:18.148973 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.148753 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:18.148973 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.148780 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:18.148973 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.148797 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:18.148973 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.148872 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce5603fa-39d7-4705-a0e3-85297ad53a78-istiod-ca-cert" pod "ce5603fa-39d7-4705-a0e3-85297ad53a78" (UID: "ce5603fa-39d7-4705-a0e3-85297ad53a78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:06:18.149260 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.149040 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-workload-certs" pod "ce5603fa-39d7-4705-a0e3-85297ad53a78" (UID: "ce5603fa-39d7-4705-a0e3-85297ad53a78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:18.151073 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.151040 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-envoy" pod "ce5603fa-39d7-4705-a0e3-85297ad53a78" (UID: "ce5603fa-39d7-4705-a0e3-85297ad53a78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:18.151181 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.151052 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-podinfo" pod "ce5603fa-39d7-4705-a0e3-85297ad53a78" (UID: "ce5603fa-39d7-4705-a0e3-85297ad53a78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:06:18.151732 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.151709 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce5603fa-39d7-4705-a0e3-85297ad53a78-kube-api-access-797c6" pod "ce5603fa-39d7-4705-a0e3-85297ad53a78" (UID: "ce5603fa-39d7-4705-a0e3-85297ad53a78"). InnerVolumeSpecName "kube-api-access-797c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:06:18.151823 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.151735 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-token" pod "ce5603fa-39d7-4705-a0e3-85297ad53a78" (UID: "ce5603fa-39d7-4705-a0e3-85297ad53a78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:06:18.249992 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.249897 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:18.249992 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.249952 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:18.249992 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.249968 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-797c6\" (UniqueName: \"kubernetes.io/projected/ce5603fa-39d7-4705-a0e3-85297ad53a78-kube-api-access-797c6\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:18.249992 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.249983 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce5603fa-39d7-4705-a0e3-85297ad53a78-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:18.250596 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.249996 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:18.250596 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.250010 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce5603fa-39d7-4705-a0e3-85297ad53a78-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:18.871212 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.871182 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" Oct 31 20:06:18.871212 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.871202 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk" event={"ID":"ce5603fa-39d7-4705-a0e3-85297ad53a78","Type":"ContainerDied","Data":"5c8da1c650868ed4932c9800cf4d5b73607befe2f9f6f03178cdabbd89cd010f"} Oct 31 20:06:18.871502 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.871269 4910 scope.go:122] "RemoveContainer" containerID="a28a4b487bc10e6a168d3b4871bfc85bd6870d6d79658c96328a29b279fb44a1" Oct 31 20:06:18.901667 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.901635 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk"] Oct 31 20:06:18.907673 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:18.907649 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-pnhwk"] Oct 31 20:06:19.754514 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:19.754475 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c34f06bd-3b50-4342-9719-bcccd4be955f" path="/var/lib/kubelet/pods/c34f06bd-3b50-4342-9719-bcccd4be955f/volumes" Oct 31 20:06:19.755018 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:19.755002 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce5603fa-39d7-4705-a0e3-85297ad53a78" path="/var/lib/kubelet/pods/ce5603fa-39d7-4705-a0e3-85297ad53a78/volumes" Oct 31 20:06:23.072514 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:23.072471 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0639dd8-6pvtb"] Oct 31 20:06:23.092974 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:23.092926 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0639dd8-6pvtb"] Oct 31 20:06:23.755963 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:23.755859 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="627b9644-f20c-4ae1-b906-f4d8f35ee8b7" path="/var/lib/kubelet/pods/627b9644-f20c-4ae1-b906-f4d8f35ee8b7/volumes" Oct 31 20:06:32.004120 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:32.004087 4910 scope.go:122] "RemoveContainer" containerID="7c55abbaca6c93f4da76038a734de400173ec59c2dd17ea756657c7fa76dab1f" Oct 31 20:06:32.014366 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:32.014346 4910 scope.go:122] "RemoveContainer" containerID="b63abc0f1d534035528a66bfdb1d88c4b4cc8f326d5fbfb4dcd57d4fc0764b00" Oct 31 20:06:32.024865 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:32.024840 4910 scope.go:122] "RemoveContainer" containerID="172f06655ec7b0a8b8b016296ac7dad8710997201c2967786dbd85926937af76" Oct 31 20:06:36.564487 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:36.564268 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt"] Oct 31 20:06:36.565021 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:36.564556 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" podUID="78d95c12-a91e-44a2-94a4-001d4c3ec257" containerName="istio-proxy" containerID="cri-o://3e4b1a16a778773e4a203fde8c7c68533c8879a4a44d61aeb201879a4a62e8a3" gracePeriod=30 Oct 31 20:06:41.863620 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:41.863584 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 20:06:41.907540 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:41.907502 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-envoy\") pod \"78d95c12-a91e-44a2-94a4-001d4c3ec257\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " Oct 31 20:06:41.907740 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:41.907586 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-workload-socket\") pod \"78d95c12-a91e-44a2-94a4-001d4c3ec257\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " Oct 31 20:06:41.907740 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:41.907636 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d95c12-a91e-44a2-94a4-001d4c3ec257-kube-api-access-q2sbq\" (UniqueName: \"kubernetes.io/projected/78d95c12-a91e-44a2-94a4-001d4c3ec257-kube-api-access-q2sbq\") pod \"78d95c12-a91e-44a2-94a4-001d4c3ec257\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " Oct 31 20:06:41.907740 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:41.907678 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-podinfo\") pod \"78d95c12-a91e-44a2-94a4-001d4c3ec257\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " Oct 31 20:06:41.907740 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:41.907728 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-credential-socket\") pod \"78d95c12-a91e-44a2-94a4-001d4c3ec257\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " Oct 31 20:06:41.907995 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:41.907759 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78d95c12-a91e-44a2-94a4-001d4c3ec257-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d95c12-a91e-44a2-94a4-001d4c3ec257-istiod-ca-cert\") pod \"78d95c12-a91e-44a2-94a4-001d4c3ec257\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " Oct 31 20:06:41.907995 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:41.907804 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-workload-certs\") pod \"78d95c12-a91e-44a2-94a4-001d4c3ec257\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " Oct 31 20:06:41.907995 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:41.907842 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-token\" (UniqueName: \"kubernetes.io/projected/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-token\") pod \"78d95c12-a91e-44a2-94a4-001d4c3ec257\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " Oct 31 20:06:41.907995 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:41.907886 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-workload-socket" pod "78d95c12-a91e-44a2-94a4-001d4c3ec257" (UID: "78d95c12-a91e-44a2-94a4-001d4c3ec257"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:41.907995 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:41.907984 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-data\") pod \"78d95c12-a91e-44a2-94a4-001d4c3ec257\" (UID: \"78d95c12-a91e-44a2-94a4-001d4c3ec257\") " Oct 31 20:06:41.908817 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:41.908268 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d95c12-a91e-44a2-94a4-001d4c3ec257-istiod-ca-cert" pod "78d95c12-a91e-44a2-94a4-001d4c3ec257" (UID: "78d95c12-a91e-44a2-94a4-001d4c3ec257"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:06:41.908817 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:41.908521 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-data" pod "78d95c12-a91e-44a2-94a4-001d4c3ec257" (UID: "78d95c12-a91e-44a2-94a4-001d4c3ec257"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:41.908817 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:41.908654 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:41.908817 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:41.908676 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:41.908817 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:41.908668 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-workload-certs" pod "78d95c12-a91e-44a2-94a4-001d4c3ec257" (UID: "78d95c12-a91e-44a2-94a4-001d4c3ec257"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:41.908817 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:41.908693 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78d95c12-a91e-44a2-94a4-001d4c3ec257-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:41.908817 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:41.908785 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-credential-socket" pod "78d95c12-a91e-44a2-94a4-001d4c3ec257" (UID: "78d95c12-a91e-44a2-94a4-001d4c3ec257"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:41.910508 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:41.910480 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d95c12-a91e-44a2-94a4-001d4c3ec257-kube-api-access-q2sbq" pod "78d95c12-a91e-44a2-94a4-001d4c3ec257" (UID: "78d95c12-a91e-44a2-94a4-001d4c3ec257"). InnerVolumeSpecName "kube-api-access-q2sbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:06:41.910637 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:41.910590 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-podinfo" pod "78d95c12-a91e-44a2-94a4-001d4c3ec257" (UID: "78d95c12-a91e-44a2-94a4-001d4c3ec257"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:06:41.910789 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:41.910766 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-token" pod "78d95c12-a91e-44a2-94a4-001d4c3ec257" (UID: "78d95c12-a91e-44a2-94a4-001d4c3ec257"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:06:41.910996 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:41.910971 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-envoy" pod "78d95c12-a91e-44a2-94a4-001d4c3ec257" (UID: "78d95c12-a91e-44a2-94a4-001d4c3ec257"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:42.010055 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:42.010017 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:42.010055 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:42.010051 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2sbq\" (UniqueName: \"kubernetes.io/projected/78d95c12-a91e-44a2-94a4-001d4c3ec257-kube-api-access-q2sbq\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:42.010055 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:42.010063 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:42.010302 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:42.010072 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:42.010302 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:42.010081 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78d95c12-a91e-44a2-94a4-001d4c3ec257-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:42.010302 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:42.010089 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78d95c12-a91e-44a2-94a4-001d4c3ec257-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:42.061301 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:42.061261 4910 generic.go:356] "Generic (PLEG): container finished" podID="78d95c12-a91e-44a2-94a4-001d4c3ec257" containerID="3e4b1a16a778773e4a203fde8c7c68533c8879a4a44d61aeb201879a4a62e8a3" exitCode=0 Oct 31 20:06:42.061512 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:42.061316 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" Oct 31 20:06:42.061512 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:42.061346 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" event={"ID":"78d95c12-a91e-44a2-94a4-001d4c3ec257","Type":"ContainerDied","Data":"3e4b1a16a778773e4a203fde8c7c68533c8879a4a44d61aeb201879a4a62e8a3"} Oct 31 20:06:42.061512 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:42.061402 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt" event={"ID":"78d95c12-a91e-44a2-94a4-001d4c3ec257","Type":"ContainerDied","Data":"f39ed49f90664aa9c89c0325bb0f3953bae3208a17a94cd8fdee5e50dced2c54"} Oct 31 20:06:42.061512 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:42.061424 4910 scope.go:122] "RemoveContainer" containerID="3e4b1a16a778773e4a203fde8c7c68533c8879a4a44d61aeb201879a4a62e8a3" Oct 31 20:06:42.072905 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:42.072883 4910 scope.go:122] "RemoveContainer" containerID="3e4b1a16a778773e4a203fde8c7c68533c8879a4a44d61aeb201879a4a62e8a3" Oct 31 20:06:42.073209 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:06:42.073191 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e4b1a16a778773e4a203fde8c7c68533c8879a4a44d61aeb201879a4a62e8a3\": container with ID starting with 3e4b1a16a778773e4a203fde8c7c68533c8879a4a44d61aeb201879a4a62e8a3 not found: ID does not exist" containerID="3e4b1a16a778773e4a203fde8c7c68533c8879a4a44d61aeb201879a4a62e8a3" Oct 31 20:06:42.073261 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:42.073218 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4b1a16a778773e4a203fde8c7c68533c8879a4a44d61aeb201879a4a62e8a3"} err="failed to get container status \"3e4b1a16a778773e4a203fde8c7c68533c8879a4a44d61aeb201879a4a62e8a3\": rpc error: code = NotFound desc = could not find container \"3e4b1a16a778773e4a203fde8c7c68533c8879a4a44d61aeb201879a4a62e8a3\": container with ID starting with 3e4b1a16a778773e4a203fde8c7c68533c8879a4a44d61aeb201879a4a62e8a3 not found: ID does not exist" Oct 31 20:06:42.090078 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:42.090016 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt"] Oct 31 20:06:42.094981 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:42.094944 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-ltmpt"] Oct 31 20:06:42.716946 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:06:42.716883 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:06:43.754495 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:43.754456 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78d95c12-a91e-44a2-94a4-001d4c3ec257" path="/var/lib/kubelet/pods/78d95c12-a91e-44a2-94a4-001d4c3ec257/volumes" Oct 31 20:06:49.562495 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:49.562457 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j"] Oct 31 20:06:49.562937 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:49.562726 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" podUID="9136da5b-c568-412a-b262-eb85bb4cf6b6" containerName="istio-proxy" containerID="cri-o://c85ca61f3d3007aa3e13f0f199caf5705557081a37e6fb41cb13462f456a7f66" gracePeriod=30 Oct 31 20:06:53.531545 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:53.531506 4910 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 20:06:53.532090 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:53.531583 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" podUID="9136da5b-c568-412a-b262-eb85bb4cf6b6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 20:06:54.836572 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:54.836497 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:06:54.987818 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:54.987783 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-token\" (UniqueName: \"kubernetes.io/projected/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-token\") pod \"9136da5b-c568-412a-b262-eb85bb4cf6b6\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " Oct 31 20:06:54.988091 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:54.987828 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-podinfo\") pod \"9136da5b-c568-412a-b262-eb85bb4cf6b6\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " Oct 31 20:06:54.988091 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:54.987855 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-workload-socket\") pod \"9136da5b-c568-412a-b262-eb85bb4cf6b6\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " Oct 31 20:06:54.988091 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:54.987891 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-envoy\") pod \"9136da5b-c568-412a-b262-eb85bb4cf6b6\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " Oct 31 20:06:54.988091 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:54.987938 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-data\") pod \"9136da5b-c568-412a-b262-eb85bb4cf6b6\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " Oct 31 20:06:54.988091 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:54.988001 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-credential-socket\") pod \"9136da5b-c568-412a-b262-eb85bb4cf6b6\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " Oct 31 20:06:54.988091 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:54.988067 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9136da5b-c568-412a-b262-eb85bb4cf6b6-kube-api-access-mpw9x\" (UniqueName: \"kubernetes.io/projected/9136da5b-c568-412a-b262-eb85bb4cf6b6-kube-api-access-mpw9x\") pod \"9136da5b-c568-412a-b262-eb85bb4cf6b6\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " Oct 31 20:06:54.988767 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:54.988098 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9136da5b-c568-412a-b262-eb85bb4cf6b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9136da5b-c568-412a-b262-eb85bb4cf6b6-istiod-ca-cert\") pod \"9136da5b-c568-412a-b262-eb85bb4cf6b6\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " Oct 31 20:06:54.988767 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:54.988193 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-workload-certs\") pod \"9136da5b-c568-412a-b262-eb85bb4cf6b6\" (UID: \"9136da5b-c568-412a-b262-eb85bb4cf6b6\") " Oct 31 20:06:54.988767 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:54.988189 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-workload-socket" pod "9136da5b-c568-412a-b262-eb85bb4cf6b6" (UID: "9136da5b-c568-412a-b262-eb85bb4cf6b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:54.989081 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:54.988980 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-data" pod "9136da5b-c568-412a-b262-eb85bb4cf6b6" (UID: "9136da5b-c568-412a-b262-eb85bb4cf6b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:54.989220 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:54.989200 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9136da5b-c568-412a-b262-eb85bb4cf6b6-istiod-ca-cert" pod "9136da5b-c568-412a-b262-eb85bb4cf6b6" (UID: "9136da5b-c568-412a-b262-eb85bb4cf6b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:06:54.989292 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:54.989208 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-credential-socket" pod "9136da5b-c568-412a-b262-eb85bb4cf6b6" (UID: "9136da5b-c568-412a-b262-eb85bb4cf6b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:54.989841 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:54.989813 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-workload-certs" pod "9136da5b-c568-412a-b262-eb85bb4cf6b6" (UID: "9136da5b-c568-412a-b262-eb85bb4cf6b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:54.990515 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:54.990491 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-token" pod "9136da5b-c568-412a-b262-eb85bb4cf6b6" (UID: "9136da5b-c568-412a-b262-eb85bb4cf6b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:06:54.990648 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:54.990633 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-envoy" pod "9136da5b-c568-412a-b262-eb85bb4cf6b6" (UID: "9136da5b-c568-412a-b262-eb85bb4cf6b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:06:54.991295 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:54.991274 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-podinfo" pod "9136da5b-c568-412a-b262-eb85bb4cf6b6" (UID: "9136da5b-c568-412a-b262-eb85bb4cf6b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:06:54.991374 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:54.991291 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9136da5b-c568-412a-b262-eb85bb4cf6b6-kube-api-access-mpw9x" pod "9136da5b-c568-412a-b262-eb85bb4cf6b6" (UID: "9136da5b-c568-412a-b262-eb85bb4cf6b6"). InnerVolumeSpecName "kube-api-access-mpw9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:06:55.097200 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:55.093781 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:55.097200 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:55.093828 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:55.097200 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:55.093844 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:55.097200 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:55.093859 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:55.097200 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:55.093872 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:55.097200 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:55.093886 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:55.097200 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:55.093898 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9136da5b-c568-412a-b262-eb85bb4cf6b6-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:55.097200 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:55.093931 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpw9x\" (UniqueName: \"kubernetes.io/projected/9136da5b-c568-412a-b262-eb85bb4cf6b6-kube-api-access-mpw9x\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:55.097200 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:55.093947 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9136da5b-c568-412a-b262-eb85bb4cf6b6-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:06:55.172235 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:55.172198 4910 generic.go:356] "Generic (PLEG): container finished" podID="9136da5b-c568-412a-b262-eb85bb4cf6b6" containerID="c85ca61f3d3007aa3e13f0f199caf5705557081a37e6fb41cb13462f456a7f66" exitCode=0 Oct 31 20:06:55.172448 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:55.172254 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" event={"ID":"9136da5b-c568-412a-b262-eb85bb4cf6b6","Type":"ContainerDied","Data":"c85ca61f3d3007aa3e13f0f199caf5705557081a37e6fb41cb13462f456a7f66"} Oct 31 20:06:55.172448 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:55.172266 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" Oct 31 20:06:55.172448 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:55.172295 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j" event={"ID":"9136da5b-c568-412a-b262-eb85bb4cf6b6","Type":"ContainerDied","Data":"865fe27c74bf59a556c58a53446fc8f27dc6c052d6fdbfe809e1a38def52fb47"} Oct 31 20:06:55.172448 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:55.172327 4910 scope.go:122] "RemoveContainer" containerID="c85ca61f3d3007aa3e13f0f199caf5705557081a37e6fb41cb13462f456a7f66" Oct 31 20:06:55.183798 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:55.183778 4910 scope.go:122] "RemoveContainer" containerID="c85ca61f3d3007aa3e13f0f199caf5705557081a37e6fb41cb13462f456a7f66" Oct 31 20:06:55.184127 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:06:55.184102 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c85ca61f3d3007aa3e13f0f199caf5705557081a37e6fb41cb13462f456a7f66\": container with ID starting with c85ca61f3d3007aa3e13f0f199caf5705557081a37e6fb41cb13462f456a7f66 not found: ID does not exist" containerID="c85ca61f3d3007aa3e13f0f199caf5705557081a37e6fb41cb13462f456a7f66" Oct 31 20:06:55.184233 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:55.184133 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c85ca61f3d3007aa3e13f0f199caf5705557081a37e6fb41cb13462f456a7f66"} err="failed to get container status \"c85ca61f3d3007aa3e13f0f199caf5705557081a37e6fb41cb13462f456a7f66\": rpc error: code = NotFound desc = could not find container \"c85ca61f3d3007aa3e13f0f199caf5705557081a37e6fb41cb13462f456a7f66\": container with ID starting with c85ca61f3d3007aa3e13f0f199caf5705557081a37e6fb41cb13462f456a7f66 not found: ID does not exist" Oct 31 20:06:55.206302 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:55.206274 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j"] Oct 31 20:06:55.222937 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:55.211471 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-kgg4j"] Oct 31 20:06:55.755634 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:06:55.755604 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9136da5b-c568-412a-b262-eb85bb4cf6b6" path="/var/lib/kubelet/pods/9136da5b-c568-412a-b262-eb85bb4cf6b6/volumes" Oct 31 20:07:05.976948 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:05.976896 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j"] Oct 31 20:07:05.977447 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:05.977177 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j" podUID="2358e6a2-66b8-44b0-99b6-52b9c75e2cfe" containerName="maas-api" containerID="cri-o://dadaf4e0f0739e690f91dbc604e2fbda5d47a968d5f4b1fba29967a63cf6e8aa" gracePeriod=30 Oct 31 20:07:06.269069 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:06.268979 4910 generic.go:356] "Generic (PLEG): container finished" podID="2358e6a2-66b8-44b0-99b6-52b9c75e2cfe" containerID="dadaf4e0f0739e690f91dbc604e2fbda5d47a968d5f4b1fba29967a63cf6e8aa" exitCode=0 Oct 31 20:07:06.269069 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:06.269053 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j" event={"ID":"2358e6a2-66b8-44b0-99b6-52b9c75e2cfe","Type":"ContainerDied","Data":"dadaf4e0f0739e690f91dbc604e2fbda5d47a968d5f4b1fba29967a63cf6e8aa"} Oct 31 20:07:06.306576 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:06.306552 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j" Oct 31 20:07:06.436059 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:06.436019 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-kube-api-access-zstlb\" (UniqueName: \"kubernetes.io/projected/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-kube-api-access-zstlb\") pod \"2358e6a2-66b8-44b0-99b6-52b9c75e2cfe\" (UID: \"2358e6a2-66b8-44b0-99b6-52b9c75e2cfe\") " Oct 31 20:07:06.436260 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:06.436076 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-metrics-certs\") pod \"2358e6a2-66b8-44b0-99b6-52b9c75e2cfe\" (UID: \"2358e6a2-66b8-44b0-99b6-52b9c75e2cfe\") " Oct 31 20:07:06.436260 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:06.436116 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-maas-api-tls\") pod \"2358e6a2-66b8-44b0-99b6-52b9c75e2cfe\" (UID: \"2358e6a2-66b8-44b0-99b6-52b9c75e2cfe\") " Oct 31 20:07:06.436402 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:06.436308 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-openshift-service-ca\") pod \"2358e6a2-66b8-44b0-99b6-52b9c75e2cfe\" (UID: \"2358e6a2-66b8-44b0-99b6-52b9c75e2cfe\") " Oct 31 20:07:06.436708 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:06.436669 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-openshift-service-ca" pod "2358e6a2-66b8-44b0-99b6-52b9c75e2cfe" (UID: "2358e6a2-66b8-44b0-99b6-52b9c75e2cfe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:07:06.436851 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:06.436787 4910 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-openshift-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:06.438477 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:06.438450 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-metrics-certs" pod "2358e6a2-66b8-44b0-99b6-52b9c75e2cfe" (UID: "2358e6a2-66b8-44b0-99b6-52b9c75e2cfe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:07:06.438626 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:06.438605 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-maas-api-tls" pod "2358e6a2-66b8-44b0-99b6-52b9c75e2cfe" (UID: "2358e6a2-66b8-44b0-99b6-52b9c75e2cfe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:07:06.440826 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:06.438728 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-kube-api-access-zstlb" pod "2358e6a2-66b8-44b0-99b6-52b9c75e2cfe" (UID: "2358e6a2-66b8-44b0-99b6-52b9c75e2cfe"). InnerVolumeSpecName "kube-api-access-zstlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:07:06.538407 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:06.538324 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zstlb\" (UniqueName: \"kubernetes.io/projected/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-kube-api-access-zstlb\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:06.538407 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:06.538357 4910 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-metrics-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:06.538407 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:06.538367 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe-maas-api-tls\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:07.278781 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:07.278749 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j" Oct 31 20:07:07.279361 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:07.278756 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j" event={"ID":"2358e6a2-66b8-44b0-99b6-52b9c75e2cfe","Type":"ContainerDied","Data":"f205c15c0265abeb91a5f0d98125c391bda43c21f02258bca9ed75e0b496e250"} Oct 31 20:07:07.279361 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:07.278884 4910 scope.go:122] "RemoveContainer" containerID="dadaf4e0f0739e690f91dbc604e2fbda5d47a968d5f4b1fba29967a63cf6e8aa" Oct 31 20:07:07.306803 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:07.306777 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j"] Oct 31 20:07:07.313604 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:07.313578 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c9b9a1-58bf4cd588-lwb9j"] Oct 31 20:07:07.755044 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:07.755013 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2358e6a2-66b8-44b0-99b6-52b9c75e2cfe" path="/var/lib/kubelet/pods/2358e6a2-66b8-44b0-99b6-52b9c75e2cfe/volumes" Oct 31 20:07:12.715445 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:07:12.715398 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:07:32.126320 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:32.126280 4910 scope.go:122] "RemoveContainer" containerID="d5ccb6440d13962054e8f787e4ce51dd9c5f794ce74f89bdfd581c19d6dda0f6" Oct 31 20:07:32.137623 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:32.137598 4910 scope.go:122] "RemoveContainer" containerID="f20adc98ae4a1f1417ecebbb0124a14ae603c17cbc51097b0ab8e4fe030696b2" Oct 31 20:07:32.147154 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:32.147135 4910 scope.go:122] "RemoveContainer" containerID="365eceb1dd10d827dec0ee7e95296d221746de55e034e06ac7cfa76d2612f3a0" Oct 31 20:07:32.487153 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:32.487063 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59"] Oct 31 20:07:32.487339 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:32.487310 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" podUID="55d3a124-18b2-4198-b3ee-7ae4180538f8" containerName="istio-proxy" containerID="cri-o://18ee071b9c433b066b559695ce61d7ba4ce7f89cbfea30442aa2cba6a19797fd" gracePeriod=30 Oct 31 20:07:37.749740 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.749713 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:07:37.835564 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.835518 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-workload-socket\") pod \"55d3a124-18b2-4198-b3ee-7ae4180538f8\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " Oct 31 20:07:37.835733 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.835606 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55d3a124-18b2-4198-b3ee-7ae4180538f8-kube-api-access-zzrnb\" (UniqueName: \"kubernetes.io/projected/55d3a124-18b2-4198-b3ee-7ae4180538f8-kube-api-access-zzrnb\") pod \"55d3a124-18b2-4198-b3ee-7ae4180538f8\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " Oct 31 20:07:37.835733 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.835674 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55d3a124-18b2-4198-b3ee-7ae4180538f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55d3a124-18b2-4198-b3ee-7ae4180538f8-istiod-ca-cert\") pod \"55d3a124-18b2-4198-b3ee-7ae4180538f8\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " Oct 31 20:07:37.835880 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.835737 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-data\") pod \"55d3a124-18b2-4198-b3ee-7ae4180538f8\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " Oct 31 20:07:37.835880 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.835791 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-token\" (UniqueName: \"kubernetes.io/projected/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-token\") pod \"55d3a124-18b2-4198-b3ee-7ae4180538f8\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " Oct 31 20:07:37.835880 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.835793 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-workload-socket" pod "55d3a124-18b2-4198-b3ee-7ae4180538f8" (UID: "55d3a124-18b2-4198-b3ee-7ae4180538f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:07:37.835880 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.835818 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-workload-certs\") pod \"55d3a124-18b2-4198-b3ee-7ae4180538f8\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " Oct 31 20:07:37.835880 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.835857 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-envoy\") pod \"55d3a124-18b2-4198-b3ee-7ae4180538f8\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " Oct 31 20:07:37.836209 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.835888 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-podinfo\") pod \"55d3a124-18b2-4198-b3ee-7ae4180538f8\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " Oct 31 20:07:37.836209 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.835962 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-credential-socket\") pod \"55d3a124-18b2-4198-b3ee-7ae4180538f8\" (UID: \"55d3a124-18b2-4198-b3ee-7ae4180538f8\") " Oct 31 20:07:37.836209 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.836042 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-data" pod "55d3a124-18b2-4198-b3ee-7ae4180538f8" (UID: "55d3a124-18b2-4198-b3ee-7ae4180538f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:07:37.836209 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.836183 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-workload-certs" pod "55d3a124-18b2-4198-b3ee-7ae4180538f8" (UID: "55d3a124-18b2-4198-b3ee-7ae4180538f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:07:37.836426 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.836219 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55d3a124-18b2-4198-b3ee-7ae4180538f8-istiod-ca-cert" pod "55d3a124-18b2-4198-b3ee-7ae4180538f8" (UID: "55d3a124-18b2-4198-b3ee-7ae4180538f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:07:37.836426 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.836391 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-credential-socket" pod "55d3a124-18b2-4198-b3ee-7ae4180538f8" (UID: "55d3a124-18b2-4198-b3ee-7ae4180538f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:07:37.836682 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.836657 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:37.836795 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.836688 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55d3a124-18b2-4198-b3ee-7ae4180538f8-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:37.836795 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.836702 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:37.836795 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.836716 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:37.836795 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.836729 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:37.838440 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.838419 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-envoy" pod "55d3a124-18b2-4198-b3ee-7ae4180538f8" (UID: "55d3a124-18b2-4198-b3ee-7ae4180538f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:07:37.838526 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.838484 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-token" pod "55d3a124-18b2-4198-b3ee-7ae4180538f8" (UID: "55d3a124-18b2-4198-b3ee-7ae4180538f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:07:37.838526 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.838515 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d3a124-18b2-4198-b3ee-7ae4180538f8-kube-api-access-zzrnb" pod "55d3a124-18b2-4198-b3ee-7ae4180538f8" (UID: "55d3a124-18b2-4198-b3ee-7ae4180538f8"). InnerVolumeSpecName "kube-api-access-zzrnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:07:37.839214 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.839195 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-podinfo" pod "55d3a124-18b2-4198-b3ee-7ae4180538f8" (UID: "55d3a124-18b2-4198-b3ee-7ae4180538f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:07:37.938015 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.937978 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzrnb\" (UniqueName: \"kubernetes.io/projected/55d3a124-18b2-4198-b3ee-7ae4180538f8-kube-api-access-zzrnb\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:37.938015 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.938008 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:37.938015 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.938021 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:37.938255 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:37.938030 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55d3a124-18b2-4198-b3ee-7ae4180538f8-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:38.524323 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:38.524275 4910 generic.go:356] "Generic (PLEG): container finished" podID="55d3a124-18b2-4198-b3ee-7ae4180538f8" containerID="18ee071b9c433b066b559695ce61d7ba4ce7f89cbfea30442aa2cba6a19797fd" exitCode=0 Oct 31 20:07:38.524566 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:38.524336 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" Oct 31 20:07:38.524566 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:38.524355 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" event={"ID":"55d3a124-18b2-4198-b3ee-7ae4180538f8","Type":"ContainerDied","Data":"18ee071b9c433b066b559695ce61d7ba4ce7f89cbfea30442aa2cba6a19797fd"} Oct 31 20:07:38.524566 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:38.524421 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59" event={"ID":"55d3a124-18b2-4198-b3ee-7ae4180538f8","Type":"ContainerDied","Data":"822c32ae02faf3582d48323ec2c05b19ae61ddb863487547450f54dc77933a02"} Oct 31 20:07:38.524566 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:38.524452 4910 scope.go:122] "RemoveContainer" containerID="18ee071b9c433b066b559695ce61d7ba4ce7f89cbfea30442aa2cba6a19797fd" Oct 31 20:07:38.540097 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:38.540075 4910 scope.go:122] "RemoveContainer" containerID="18ee071b9c433b066b559695ce61d7ba4ce7f89cbfea30442aa2cba6a19797fd" Oct 31 20:07:38.540513 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:07:38.540495 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18ee071b9c433b066b559695ce61d7ba4ce7f89cbfea30442aa2cba6a19797fd\": container with ID starting with 18ee071b9c433b066b559695ce61d7ba4ce7f89cbfea30442aa2cba6a19797fd not found: ID does not exist" containerID="18ee071b9c433b066b559695ce61d7ba4ce7f89cbfea30442aa2cba6a19797fd" Oct 31 20:07:38.540601 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:38.540528 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ee071b9c433b066b559695ce61d7ba4ce7f89cbfea30442aa2cba6a19797fd"} err="failed to get container status \"18ee071b9c433b066b559695ce61d7ba4ce7f89cbfea30442aa2cba6a19797fd\": rpc error: code = NotFound desc = could not find container \"18ee071b9c433b066b559695ce61d7ba4ce7f89cbfea30442aa2cba6a19797fd\": container with ID starting with 18ee071b9c433b066b559695ce61d7ba4ce7f89cbfea30442aa2cba6a19797fd not found: ID does not exist" Oct 31 20:07:38.570741 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:38.570708 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59"] Oct 31 20:07:38.577055 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:38.577028 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-rsj59"] Oct 31 20:07:38.854947 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:38.854894 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8"] Oct 31 20:07:38.855340 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:38.855146 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" podUID="15e605d1-e3b9-4c0b-9b03-689c480dd89c" containerName="istio-proxy" containerID="cri-o://ad8bda9ae5e5870b65111ae1b0a4b5d912f159710737aa492fe40a8bb81bbb57" gracePeriod=30 Oct 31 20:07:39.755278 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:39.755245 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55d3a124-18b2-4198-b3ee-7ae4180538f8" path="/var/lib/kubelet/pods/55d3a124-18b2-4198-b3ee-7ae4180538f8/volumes" Oct 31 20:07:42.712184 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:07:42.712145 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:07:44.143652 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.143617 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 20:07:44.315481 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.315440 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-envoy\") pod \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " Oct 31 20:07:44.315648 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.315506 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-podinfo\") pod \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " Oct 31 20:07:44.315648 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.315556 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-workload-socket\") pod \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " Oct 31 20:07:44.315648 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.315594 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-workload-certs\") pod \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " Oct 31 20:07:44.315799 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.315745 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istiod-ca-cert\") pod \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " Oct 31 20:07:44.315799 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.315784 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-data\") pod \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " Oct 31 20:07:44.315947 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.315817 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-credential-socket\") pod \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " Oct 31 20:07:44.315947 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.315847 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-token\" (UniqueName: \"kubernetes.io/projected/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-token\") pod \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " Oct 31 20:07:44.315947 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.315885 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15e605d1-e3b9-4c0b-9b03-689c480dd89c-kube-api-access-l5fq4\" (UniqueName: \"kubernetes.io/projected/15e605d1-e3b9-4c0b-9b03-689c480dd89c-kube-api-access-l5fq4\") pod \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\" (UID: \"15e605d1-e3b9-4c0b-9b03-689c480dd89c\") " Oct 31 20:07:44.316217 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.315940 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-workload-certs" pod "15e605d1-e3b9-4c0b-9b03-689c480dd89c" (UID: "15e605d1-e3b9-4c0b-9b03-689c480dd89c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:07:44.316217 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.315976 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-workload-socket" pod "15e605d1-e3b9-4c0b-9b03-689c480dd89c" (UID: "15e605d1-e3b9-4c0b-9b03-689c480dd89c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:07:44.316217 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.316117 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-credential-socket" pod "15e605d1-e3b9-4c0b-9b03-689c480dd89c" (UID: "15e605d1-e3b9-4c0b-9b03-689c480dd89c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:07:44.316217 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.316174 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-data" pod "15e605d1-e3b9-4c0b-9b03-689c480dd89c" (UID: "15e605d1-e3b9-4c0b-9b03-689c480dd89c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:07:44.316217 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.316176 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istiod-ca-cert" pod "15e605d1-e3b9-4c0b-9b03-689c480dd89c" (UID: "15e605d1-e3b9-4c0b-9b03-689c480dd89c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:07:44.316487 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.316349 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:44.316487 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.316372 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:44.316487 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.316388 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:44.316487 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.316402 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:44.316487 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.316417 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:44.318602 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.318243 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-token" pod "15e605d1-e3b9-4c0b-9b03-689c480dd89c" (UID: "15e605d1-e3b9-4c0b-9b03-689c480dd89c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:07:44.318602 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.318353 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-envoy" pod "15e605d1-e3b9-4c0b-9b03-689c480dd89c" (UID: "15e605d1-e3b9-4c0b-9b03-689c480dd89c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:07:44.318602 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.318520 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-podinfo" pod "15e605d1-e3b9-4c0b-9b03-689c480dd89c" (UID: "15e605d1-e3b9-4c0b-9b03-689c480dd89c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:07:44.318850 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.318624 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e605d1-e3b9-4c0b-9b03-689c480dd89c-kube-api-access-l5fq4" pod "15e605d1-e3b9-4c0b-9b03-689c480dd89c" (UID: "15e605d1-e3b9-4c0b-9b03-689c480dd89c"). InnerVolumeSpecName "kube-api-access-l5fq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:07:44.418054 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.417960 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:44.418054 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.417999 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5fq4\" (UniqueName: \"kubernetes.io/projected/15e605d1-e3b9-4c0b-9b03-689c480dd89c-kube-api-access-l5fq4\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:44.418054 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.418014 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:44.418054 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.418028 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15e605d1-e3b9-4c0b-9b03-689c480dd89c-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:07:44.574018 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.573982 4910 generic.go:356] "Generic (PLEG): container finished" podID="15e605d1-e3b9-4c0b-9b03-689c480dd89c" containerID="ad8bda9ae5e5870b65111ae1b0a4b5d912f159710737aa492fe40a8bb81bbb57" exitCode=0 Oct 31 20:07:44.574197 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.574031 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" Oct 31 20:07:44.574197 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.574063 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" event={"ID":"15e605d1-e3b9-4c0b-9b03-689c480dd89c","Type":"ContainerDied","Data":"ad8bda9ae5e5870b65111ae1b0a4b5d912f159710737aa492fe40a8bb81bbb57"} Oct 31 20:07:44.574197 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.574121 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8" event={"ID":"15e605d1-e3b9-4c0b-9b03-689c480dd89c","Type":"ContainerDied","Data":"19e03dca9ecd33ed64066e697c92732065b9e1d727d55074cc081560271d4a2a"} Oct 31 20:07:44.574197 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.574149 4910 scope.go:122] "RemoveContainer" containerID="ad8bda9ae5e5870b65111ae1b0a4b5d912f159710737aa492fe40a8bb81bbb57" Oct 31 20:07:44.584847 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.584828 4910 scope.go:122] "RemoveContainer" containerID="ad8bda9ae5e5870b65111ae1b0a4b5d912f159710737aa492fe40a8bb81bbb57" Oct 31 20:07:44.585144 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:07:44.585125 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad8bda9ae5e5870b65111ae1b0a4b5d912f159710737aa492fe40a8bb81bbb57\": container with ID starting with ad8bda9ae5e5870b65111ae1b0a4b5d912f159710737aa492fe40a8bb81bbb57 not found: ID does not exist" containerID="ad8bda9ae5e5870b65111ae1b0a4b5d912f159710737aa492fe40a8bb81bbb57" Oct 31 20:07:44.585198 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.585153 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad8bda9ae5e5870b65111ae1b0a4b5d912f159710737aa492fe40a8bb81bbb57"} err="failed to get container status \"ad8bda9ae5e5870b65111ae1b0a4b5d912f159710737aa492fe40a8bb81bbb57\": rpc error: code = NotFound desc = could not find container \"ad8bda9ae5e5870b65111ae1b0a4b5d912f159710737aa492fe40a8bb81bbb57\": container with ID starting with ad8bda9ae5e5870b65111ae1b0a4b5d912f159710737aa492fe40a8bb81bbb57 not found: ID does not exist" Oct 31 20:07:44.605677 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.605651 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8"] Oct 31 20:07:44.608327 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:44.608305 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-5g7p8"] Oct 31 20:07:45.753979 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:45.753947 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15e605d1-e3b9-4c0b-9b03-689c480dd89c" path="/var/lib/kubelet/pods/15e605d1-e3b9-4c0b-9b03-689c480dd89c/volumes" Oct 31 20:07:56.601959 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:56.600115 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29"] Oct 31 20:07:56.601959 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:07:56.600477 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" podUID="f1917335-8922-4b66-a5c4-ec463de8e3ca" containerName="istio-proxy" containerID="cri-o://61780a0fbfc07598cfcac6d015ab903cfcf8b87e0cac08f11fae503401e0f683" gracePeriod=30 Oct 31 20:08:01.205578 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:01.205543 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45"] Oct 31 20:08:01.206170 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:01.205765 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" podUID="0da37695-41cb-4698-9658-c5f7eb993220" containerName="istio-proxy" containerID="cri-o://712c27d054fe6dd01c9c3772672019bec42c0ef4a1c320afbb09566c7166d5d7" gracePeriod=30 Oct 31 20:08:01.885417 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:01.885355 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 20:08:01.931328 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:01.931290 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-podinfo\") pod \"f1917335-8922-4b66-a5c4-ec463de8e3ca\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " Oct 31 20:08:01.931530 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:01.931342 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1917335-8922-4b66-a5c4-ec463de8e3ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1917335-8922-4b66-a5c4-ec463de8e3ca-istiod-ca-cert\") pod \"f1917335-8922-4b66-a5c4-ec463de8e3ca\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " Oct 31 20:08:01.931530 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:01.931405 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-envoy\") pod \"f1917335-8922-4b66-a5c4-ec463de8e3ca\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " Oct 31 20:08:01.931530 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:01.931444 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-data\") pod \"f1917335-8922-4b66-a5c4-ec463de8e3ca\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " Oct 31 20:08:01.931530 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:01.931475 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-workload-socket\") pod \"f1917335-8922-4b66-a5c4-ec463de8e3ca\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " Oct 31 20:08:01.931530 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:01.931514 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1917335-8922-4b66-a5c4-ec463de8e3ca-kube-api-access-847qx\" (UniqueName: \"kubernetes.io/projected/f1917335-8922-4b66-a5c4-ec463de8e3ca-kube-api-access-847qx\") pod \"f1917335-8922-4b66-a5c4-ec463de8e3ca\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " Oct 31 20:08:01.931844 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:01.931550 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-workload-certs\") pod \"f1917335-8922-4b66-a5c4-ec463de8e3ca\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " Oct 31 20:08:01.931844 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:01.931581 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-credential-socket\") pod \"f1917335-8922-4b66-a5c4-ec463de8e3ca\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " Oct 31 20:08:01.931844 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:01.931636 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-token\" (UniqueName: \"kubernetes.io/projected/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-token\") pod \"f1917335-8922-4b66-a5c4-ec463de8e3ca\" (UID: \"f1917335-8922-4b66-a5c4-ec463de8e3ca\") " Oct 31 20:08:01.931844 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:01.931816 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1917335-8922-4b66-a5c4-ec463de8e3ca-istiod-ca-cert" pod "f1917335-8922-4b66-a5c4-ec463de8e3ca" (UID: "f1917335-8922-4b66-a5c4-ec463de8e3ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:08:01.932184 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:01.931861 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-workload-socket" pod "f1917335-8922-4b66-a5c4-ec463de8e3ca" (UID: "f1917335-8922-4b66-a5c4-ec463de8e3ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:01.932357 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:01.932333 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1917335-8922-4b66-a5c4-ec463de8e3ca-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:01.932447 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:01.932365 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:01.933160 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:01.933133 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-workload-certs" pod "f1917335-8922-4b66-a5c4-ec463de8e3ca" (UID: "f1917335-8922-4b66-a5c4-ec463de8e3ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:01.933511 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:01.933470 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-credential-socket" pod "f1917335-8922-4b66-a5c4-ec463de8e3ca" (UID: "f1917335-8922-4b66-a5c4-ec463de8e3ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:01.933722 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:01.933683 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-data" pod "f1917335-8922-4b66-a5c4-ec463de8e3ca" (UID: "f1917335-8922-4b66-a5c4-ec463de8e3ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:01.935142 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:01.935112 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-token" pod "f1917335-8922-4b66-a5c4-ec463de8e3ca" (UID: "f1917335-8922-4b66-a5c4-ec463de8e3ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:08:01.935421 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:01.935400 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-envoy" pod "f1917335-8922-4b66-a5c4-ec463de8e3ca" (UID: "f1917335-8922-4b66-a5c4-ec463de8e3ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:01.936116 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:01.936088 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1917335-8922-4b66-a5c4-ec463de8e3ca-kube-api-access-847qx" pod "f1917335-8922-4b66-a5c4-ec463de8e3ca" (UID: "f1917335-8922-4b66-a5c4-ec463de8e3ca"). InnerVolumeSpecName "kube-api-access-847qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:08:01.937012 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:01.936987 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-podinfo" pod "f1917335-8922-4b66-a5c4-ec463de8e3ca" (UID: "f1917335-8922-4b66-a5c4-ec463de8e3ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:08:02.034009 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:02.033967 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-847qx\" (UniqueName: \"kubernetes.io/projected/f1917335-8922-4b66-a5c4-ec463de8e3ca-kube-api-access-847qx\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:02.034009 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:02.033998 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:02.034009 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:02.034011 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:02.034353 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:02.034026 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:02.034353 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:02.034035 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:02.034353 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:02.034045 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:02.034353 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:02.034056 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1917335-8922-4b66-a5c4-ec463de8e3ca-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:02.094027 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:02.093021 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp"] Oct 31 20:08:02.101976 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:02.101945 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6"] Oct 31 20:08:02.703591 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:02.703555 4910 generic.go:356] "Generic (PLEG): container finished" podID="f1917335-8922-4b66-a5c4-ec463de8e3ca" containerID="61780a0fbfc07598cfcac6d015ab903cfcf8b87e0cac08f11fae503401e0f683" exitCode=0 Oct 31 20:08:02.704107 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:02.703604 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" event={"ID":"f1917335-8922-4b66-a5c4-ec463de8e3ca","Type":"ContainerDied","Data":"61780a0fbfc07598cfcac6d015ab903cfcf8b87e0cac08f11fae503401e0f683"} Oct 31 20:08:02.704107 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:02.703635 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" Oct 31 20:08:02.704107 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:02.703670 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29" event={"ID":"f1917335-8922-4b66-a5c4-ec463de8e3ca","Type":"ContainerDied","Data":"48e74265626f88861c89a9d80a385a9e6333a65317bba47fb3d02d209dbed9d9"} Oct 31 20:08:02.704107 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:02.703692 4910 scope.go:122] "RemoveContainer" containerID="61780a0fbfc07598cfcac6d015ab903cfcf8b87e0cac08f11fae503401e0f683" Oct 31 20:08:02.715505 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:02.715483 4910 scope.go:122] "RemoveContainer" containerID="61780a0fbfc07598cfcac6d015ab903cfcf8b87e0cac08f11fae503401e0f683" Oct 31 20:08:02.715835 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:08:02.715808 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61780a0fbfc07598cfcac6d015ab903cfcf8b87e0cac08f11fae503401e0f683\": container with ID starting with 61780a0fbfc07598cfcac6d015ab903cfcf8b87e0cac08f11fae503401e0f683 not found: ID does not exist" containerID="61780a0fbfc07598cfcac6d015ab903cfcf8b87e0cac08f11fae503401e0f683" Oct 31 20:08:02.715980 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:02.715846 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61780a0fbfc07598cfcac6d015ab903cfcf8b87e0cac08f11fae503401e0f683"} err="failed to get container status \"61780a0fbfc07598cfcac6d015ab903cfcf8b87e0cac08f11fae503401e0f683\": rpc error: code = NotFound desc = could not find container \"61780a0fbfc07598cfcac6d015ab903cfcf8b87e0cac08f11fae503401e0f683\": container with ID starting with 61780a0fbfc07598cfcac6d015ab903cfcf8b87e0cac08f11fae503401e0f683 not found: ID does not exist" Oct 31 20:08:02.741749 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:02.741720 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29"] Oct 31 20:08:02.754874 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:02.754846 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-2fp29"] Oct 31 20:08:03.753704 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:03.753669 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1917335-8922-4b66-a5c4-ec463de8e3ca" path="/var/lib/kubelet/pods/f1917335-8922-4b66-a5c4-ec463de8e3ca/volumes" Oct 31 20:08:06.474775 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.474712 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 20:08:06.581427 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.581391 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0da37695-41cb-4698-9658-c5f7eb993220-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da37695-41cb-4698-9658-c5f7eb993220-istio-podinfo\") pod \"0da37695-41cb-4698-9658-c5f7eb993220\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " Oct 31 20:08:06.581427 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.581434 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-istio-envoy\") pod \"0da37695-41cb-4698-9658-c5f7eb993220\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " Oct 31 20:08:06.581672 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.581456 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-credential-socket\") pod \"0da37695-41cb-4698-9658-c5f7eb993220\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " Oct 31 20:08:06.581672 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.581498 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da37695-41cb-4698-9658-c5f7eb993220-istio-token\" (UniqueName: \"kubernetes.io/projected/0da37695-41cb-4698-9658-c5f7eb993220-istio-token\") pod \"0da37695-41cb-4698-9658-c5f7eb993220\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " Oct 31 20:08:06.581672 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.581558 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-workload-socket\") pod \"0da37695-41cb-4698-9658-c5f7eb993220\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " Oct 31 20:08:06.581672 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.581586 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-istio-data\") pod \"0da37695-41cb-4698-9658-c5f7eb993220\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " Oct 31 20:08:06.581945 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.581669 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-workload-certs\") pod \"0da37695-41cb-4698-9658-c5f7eb993220\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " Oct 31 20:08:06.581945 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.581726 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da37695-41cb-4698-9658-c5f7eb993220-kube-api-access-5fg69\" (UniqueName: \"kubernetes.io/projected/0da37695-41cb-4698-9658-c5f7eb993220-kube-api-access-5fg69\") pod \"0da37695-41cb-4698-9658-c5f7eb993220\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " Oct 31 20:08:06.581945 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.581766 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da37695-41cb-4698-9658-c5f7eb993220-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da37695-41cb-4698-9658-c5f7eb993220-istiod-ca-cert\") pod \"0da37695-41cb-4698-9658-c5f7eb993220\" (UID: \"0da37695-41cb-4698-9658-c5f7eb993220\") " Oct 31 20:08:06.581945 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.581879 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-workload-socket" pod "0da37695-41cb-4698-9658-c5f7eb993220" (UID: "0da37695-41cb-4698-9658-c5f7eb993220"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:06.582244 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.582070 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-workload-certs" pod "0da37695-41cb-4698-9658-c5f7eb993220" (UID: "0da37695-41cb-4698-9658-c5f7eb993220"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:06.582244 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.582087 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-istio-data" pod "0da37695-41cb-4698-9658-c5f7eb993220" (UID: "0da37695-41cb-4698-9658-c5f7eb993220"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:06.582244 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.582169 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-credential-socket" pod "0da37695-41cb-4698-9658-c5f7eb993220" (UID: "0da37695-41cb-4698-9658-c5f7eb993220"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:06.582244 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.582199 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da37695-41cb-4698-9658-c5f7eb993220-istiod-ca-cert" pod "0da37695-41cb-4698-9658-c5f7eb993220" (UID: "0da37695-41cb-4698-9658-c5f7eb993220"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:08:06.582467 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.582280 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:06.582467 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.582296 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:06.582467 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.582310 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:06.582467 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.582319 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da37695-41cb-4698-9658-c5f7eb993220-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:06.582467 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.582327 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:06.584197 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.584171 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-istio-envoy" pod "0da37695-41cb-4698-9658-c5f7eb993220" (UID: "0da37695-41cb-4698-9658-c5f7eb993220"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:06.584336 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.584195 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da37695-41cb-4698-9658-c5f7eb993220-istio-token" pod "0da37695-41cb-4698-9658-c5f7eb993220" (UID: "0da37695-41cb-4698-9658-c5f7eb993220"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:08:06.584336 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.584212 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0da37695-41cb-4698-9658-c5f7eb993220-istio-podinfo" pod "0da37695-41cb-4698-9658-c5f7eb993220" (UID: "0da37695-41cb-4698-9658-c5f7eb993220"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:08:06.584624 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.584603 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da37695-41cb-4698-9658-c5f7eb993220-kube-api-access-5fg69" pod "0da37695-41cb-4698-9658-c5f7eb993220" (UID: "0da37695-41cb-4698-9658-c5f7eb993220"). InnerVolumeSpecName "kube-api-access-5fg69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:08:06.683356 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.683317 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da37695-41cb-4698-9658-c5f7eb993220-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:06.683356 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.683349 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da37695-41cb-4698-9658-c5f7eb993220-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:06.683356 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.683363 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da37695-41cb-4698-9658-c5f7eb993220-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:06.683628 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.683376 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fg69\" (UniqueName: \"kubernetes.io/projected/0da37695-41cb-4698-9658-c5f7eb993220-kube-api-access-5fg69\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:06.735050 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.734964 4910 generic.go:356] "Generic (PLEG): container finished" podID="0da37695-41cb-4698-9658-c5f7eb993220" containerID="712c27d054fe6dd01c9c3772672019bec42c0ef4a1c320afbb09566c7166d5d7" exitCode=0 Oct 31 20:08:06.735050 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.735023 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" Oct 31 20:08:06.735301 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.735051 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" event={"ID":"0da37695-41cb-4698-9658-c5f7eb993220","Type":"ContainerDied","Data":"712c27d054fe6dd01c9c3772672019bec42c0ef4a1c320afbb09566c7166d5d7"} Oct 31 20:08:06.735301 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.735119 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45" event={"ID":"0da37695-41cb-4698-9658-c5f7eb993220","Type":"ContainerDied","Data":"6ca579c81973f4eb2ce857bfbb3a91ddf06b9cd478e1353f4a667d6bdf37702d"} Oct 31 20:08:06.735301 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.735149 4910 scope.go:122] "RemoveContainer" containerID="712c27d054fe6dd01c9c3772672019bec42c0ef4a1c320afbb09566c7166d5d7" Oct 31 20:08:06.746329 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.746309 4910 scope.go:122] "RemoveContainer" containerID="712c27d054fe6dd01c9c3772672019bec42c0ef4a1c320afbb09566c7166d5d7" Oct 31 20:08:06.746607 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:08:06.746586 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"712c27d054fe6dd01c9c3772672019bec42c0ef4a1c320afbb09566c7166d5d7\": container with ID starting with 712c27d054fe6dd01c9c3772672019bec42c0ef4a1c320afbb09566c7166d5d7 not found: ID does not exist" containerID="712c27d054fe6dd01c9c3772672019bec42c0ef4a1c320afbb09566c7166d5d7" Oct 31 20:08:06.746653 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.746619 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712c27d054fe6dd01c9c3772672019bec42c0ef4a1c320afbb09566c7166d5d7"} err="failed to get container status \"712c27d054fe6dd01c9c3772672019bec42c0ef4a1c320afbb09566c7166d5d7\": rpc error: code = NotFound desc = could not find container \"712c27d054fe6dd01c9c3772672019bec42c0ef4a1c320afbb09566c7166d5d7\": container with ID starting with 712c27d054fe6dd01c9c3772672019bec42c0ef4a1c320afbb09566c7166d5d7 not found: ID does not exist" Oct 31 20:08:06.765525 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.765500 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45"] Oct 31 20:08:06.769618 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.769592 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-86d45"] Oct 31 20:08:06.776820 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.776794 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n"] Oct 31 20:08:06.795617 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.795587 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws"] Oct 31 20:08:06.827403 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:06.827368 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb"] Oct 31 20:08:07.759094 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:07.759056 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da37695-41cb-4698-9658-c5f7eb993220" path="/var/lib/kubelet/pods/0da37695-41cb-4698-9658-c5f7eb993220/volumes" Oct 31 20:08:12.244071 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:12.244039 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz"] Oct 31 20:08:12.713554 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:08:12.713517 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:08:17.120696 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.120619 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" containerID="cri-o://ff23d0413aa10b5e6e07e70fb0fda679a07d908a0c6fc9553919fa14cb065060" gracePeriod=45 Oct 31 20:08:17.121693 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.121662 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" containerID="cri-o://02b678ebf80d8fe3f9fe3af2cd19f227363957a991e1a4b930de18df07b19454" gracePeriod=45 Oct 31 20:08:17.244940 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.244836 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz" podUID="9a4949e1-4900-4ce8-9734-ce6e73bef410" containerName="payload-pre-processing" containerID="cri-o://467a63236c40f6886d5f0c36f5079efa32cbc036ae0bdeefb18b4c0e4007267b" gracePeriod=25 Oct 31 20:08:17.373690 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.373621 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 20:08:17.410028 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.410003 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 20:08:17.511820 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.511783 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-home\" (UniqueName: \"kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-home\") pod \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\" (UID: \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\") " Oct 31 20:08:17.512060 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.511837 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-model-cache\") pod \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\" (UID: \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\") " Oct 31 20:08:17.512060 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.511872 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-tmp-dir\") pod \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\" (UID: \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\") " Oct 31 20:08:17.512060 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.511922 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d01fa9ef-252a-417c-b6d8-27375c4e1b90-tls-certs\" (UniqueName: \"kubernetes.io/secret/d01fa9ef-252a-417c-b6d8-27375c4e1b90-tls-certs\") pod \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\" (UID: \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\") " Oct 31 20:08:17.512060 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.511976 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-dshm\") pod \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\" (UID: \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\") " Oct 31 20:08:17.512060 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.512010 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cfae09f-0cde-4452-9ac1-babca8bb72a6-kube-api-access-qx5kk\" (UniqueName: \"kubernetes.io/projected/0cfae09f-0cde-4452-9ac1-babca8bb72a6-kube-api-access-qx5kk\") pod \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\" (UID: \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\") " Oct 31 20:08:17.512060 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.512057 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-dshm\") pod \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\" (UID: \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\") " Oct 31 20:08:17.512445 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.512108 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d01fa9ef-252a-417c-b6d8-27375c4e1b90-kube-api-access-m8ghh\" (UniqueName: \"kubernetes.io/projected/d01fa9ef-252a-417c-b6d8-27375c4e1b90-kube-api-access-m8ghh\") pod \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\" (UID: \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\") " Oct 31 20:08:17.512445 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.512160 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-model-cache\") pod \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\" (UID: \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\") " Oct 31 20:08:17.512445 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.512213 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-tmp-dir" pod "0cfae09f-0cde-4452-9ac1-babca8bb72a6" (UID: "0cfae09f-0cde-4452-9ac1-babca8bb72a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:17.512445 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.512229 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-tmp-dir\") pod \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\" (UID: \"d01fa9ef-252a-417c-b6d8-27375c4e1b90\") " Oct 31 20:08:17.512445 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.512280 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-home\" (UniqueName: \"kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-home\") pod \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\" (UID: \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\") " Oct 31 20:08:17.512445 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.512309 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cfae09f-0cde-4452-9ac1-babca8bb72a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/0cfae09f-0cde-4452-9ac1-babca8bb72a6-tls-certs\") pod \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\" (UID: \"0cfae09f-0cde-4452-9ac1-babca8bb72a6\") " Oct 31 20:08:17.512790 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.512510 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-home" pod "d01fa9ef-252a-417c-b6d8-27375c4e1b90" (UID: "d01fa9ef-252a-417c-b6d8-27375c4e1b90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:17.512851 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.512824 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-tmp-dir\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:17.512930 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.512851 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-home\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:17.513005 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.512970 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-model-cache" pod "0cfae09f-0cde-4452-9ac1-babca8bb72a6" (UID: "0cfae09f-0cde-4452-9ac1-babca8bb72a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:17.513153 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.513129 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-tmp-dir" pod "d01fa9ef-252a-417c-b6d8-27375c4e1b90" (UID: "d01fa9ef-252a-417c-b6d8-27375c4e1b90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:17.513402 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.513373 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-home" pod "0cfae09f-0cde-4452-9ac1-babca8bb72a6" (UID: "0cfae09f-0cde-4452-9ac1-babca8bb72a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:17.513503 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.513447 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-model-cache" pod "d01fa9ef-252a-417c-b6d8-27375c4e1b90" (UID: "d01fa9ef-252a-417c-b6d8-27375c4e1b90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:17.515466 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.515442 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cfae09f-0cde-4452-9ac1-babca8bb72a6-tls-certs" pod "0cfae09f-0cde-4452-9ac1-babca8bb72a6" (UID: "0cfae09f-0cde-4452-9ac1-babca8bb72a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:08:17.515735 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.515706 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-dshm" pod "d01fa9ef-252a-417c-b6d8-27375c4e1b90" (UID: "d01fa9ef-252a-417c-b6d8-27375c4e1b90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:17.515864 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.515808 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-dshm" pod "0cfae09f-0cde-4452-9ac1-babca8bb72a6" (UID: "0cfae09f-0cde-4452-9ac1-babca8bb72a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:17.516137 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.516116 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d01fa9ef-252a-417c-b6d8-27375c4e1b90-kube-api-access-m8ghh" pod "d01fa9ef-252a-417c-b6d8-27375c4e1b90" (UID: "d01fa9ef-252a-417c-b6d8-27375c4e1b90"). InnerVolumeSpecName "kube-api-access-m8ghh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:08:17.516504 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.516477 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d01fa9ef-252a-417c-b6d8-27375c4e1b90-tls-certs" pod "d01fa9ef-252a-417c-b6d8-27375c4e1b90" (UID: "d01fa9ef-252a-417c-b6d8-27375c4e1b90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:08:17.516603 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.516520 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cfae09f-0cde-4452-9ac1-babca8bb72a6-kube-api-access-qx5kk" pod "0cfae09f-0cde-4452-9ac1-babca8bb72a6" (UID: "0cfae09f-0cde-4452-9ac1-babca8bb72a6"). InnerVolumeSpecName "kube-api-access-qx5kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:08:17.614137 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.614098 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-model-cache\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:17.614137 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.614129 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d01fa9ef-252a-417c-b6d8-27375c4e1b90-tls-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:17.614137 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.614139 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-dshm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:17.614396 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.614151 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx5kk\" (UniqueName: \"kubernetes.io/projected/0cfae09f-0cde-4452-9ac1-babca8bb72a6-kube-api-access-qx5kk\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:17.614396 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.614167 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-dshm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:17.614396 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.614175 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8ghh\" (UniqueName: \"kubernetes.io/projected/d01fa9ef-252a-417c-b6d8-27375c4e1b90-kube-api-access-m8ghh\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:17.614396 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.614183 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-model-cache\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:17.614396 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.614192 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d01fa9ef-252a-417c-b6d8-27375c4e1b90-tmp-dir\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:17.614396 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.614201 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cfae09f-0cde-4452-9ac1-babca8bb72a6-home\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:17.614396 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.614208 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cfae09f-0cde-4452-9ac1-babca8bb72a6-tls-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:17.806800 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.806762 4910 generic.go:356] "Generic (PLEG): container finished" podID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerID="02b678ebf80d8fe3f9fe3af2cd19f227363957a991e1a4b930de18df07b19454" exitCode=0 Oct 31 20:08:17.807013 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.806805 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" Oct 31 20:08:17.807013 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.806834 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" event={"ID":"0cfae09f-0cde-4452-9ac1-babca8bb72a6","Type":"ContainerDied","Data":"02b678ebf80d8fe3f9fe3af2cd19f227363957a991e1a4b930de18df07b19454"} Oct 31 20:08:17.807013 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.806877 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" event={"ID":"0cfae09f-0cde-4452-9ac1-babca8bb72a6","Type":"ContainerDied","Data":"ff66440d61bfa84bb56fa794830450f285bdf43bf075ae0cdb836809ebe09155"} Oct 31 20:08:17.807013 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.806897 4910 scope.go:122] "RemoveContainer" containerID="02b678ebf80d8fe3f9fe3af2cd19f227363957a991e1a4b930de18df07b19454" Oct 31 20:08:17.808646 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.808623 4910 generic.go:356] "Generic (PLEG): container finished" podID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerID="ff23d0413aa10b5e6e07e70fb0fda679a07d908a0c6fc9553919fa14cb065060" exitCode=0 Oct 31 20:08:17.808785 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.808660 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" event={"ID":"d01fa9ef-252a-417c-b6d8-27375c4e1b90","Type":"ContainerDied","Data":"ff23d0413aa10b5e6e07e70fb0fda679a07d908a0c6fc9553919fa14cb065060"} Oct 31 20:08:17.808785 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.808689 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" event={"ID":"d01fa9ef-252a-417c-b6d8-27375c4e1b90","Type":"ContainerDied","Data":"db44931ad4d177424c21bf97030ccd5c747df2a1705fbe9fcb9706447f5333ef"} Oct 31 20:08:17.808785 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.808745 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp" Oct 31 20:08:17.827438 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.827409 4910 scope.go:122] "RemoveContainer" containerID="02b678ebf80d8fe3f9fe3af2cd19f227363957a991e1a4b930de18df07b19454" Oct 31 20:08:17.828410 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:08:17.828378 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02b678ebf80d8fe3f9fe3af2cd19f227363957a991e1a4b930de18df07b19454\": container with ID starting with 02b678ebf80d8fe3f9fe3af2cd19f227363957a991e1a4b930de18df07b19454 not found: ID does not exist" containerID="02b678ebf80d8fe3f9fe3af2cd19f227363957a991e1a4b930de18df07b19454" Oct 31 20:08:17.828613 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.828573 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02b678ebf80d8fe3f9fe3af2cd19f227363957a991e1a4b930de18df07b19454"} err="failed to get container status \"02b678ebf80d8fe3f9fe3af2cd19f227363957a991e1a4b930de18df07b19454\": rpc error: code = NotFound desc = could not find container \"02b678ebf80d8fe3f9fe3af2cd19f227363957a991e1a4b930de18df07b19454\": container with ID starting with 02b678ebf80d8fe3f9fe3af2cd19f227363957a991e1a4b930de18df07b19454 not found: ID does not exist" Oct 31 20:08:17.828613 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.828608 4910 scope.go:122] "RemoveContainer" containerID="ff23d0413aa10b5e6e07e70fb0fda679a07d908a0c6fc9553919fa14cb065060" Oct 31 20:08:17.838178 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.838154 4910 scope.go:122] "RemoveContainer" containerID="ff23d0413aa10b5e6e07e70fb0fda679a07d908a0c6fc9553919fa14cb065060" Oct 31 20:08:17.838452 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:08:17.838428 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff23d0413aa10b5e6e07e70fb0fda679a07d908a0c6fc9553919fa14cb065060\": container with ID starting with ff23d0413aa10b5e6e07e70fb0fda679a07d908a0c6fc9553919fa14cb065060 not found: ID does not exist" containerID="ff23d0413aa10b5e6e07e70fb0fda679a07d908a0c6fc9553919fa14cb065060" Oct 31 20:08:17.838504 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.838464 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff23d0413aa10b5e6e07e70fb0fda679a07d908a0c6fc9553919fa14cb065060"} err="failed to get container status \"ff23d0413aa10b5e6e07e70fb0fda679a07d908a0c6fc9553919fa14cb065060\": rpc error: code = NotFound desc = could not find container \"ff23d0413aa10b5e6e07e70fb0fda679a07d908a0c6fc9553919fa14cb065060\": container with ID starting with ff23d0413aa10b5e6e07e70fb0fda679a07d908a0c6fc9553919fa14cb065060 not found: ID does not exist" Oct 31 20:08:17.846534 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.846510 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp"] Oct 31 20:08:17.867119 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.867093 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-8b083a/premium-w1-8b083a-kserve-f79655c75-5c7gp"] Oct 31 20:08:17.893565 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.893535 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6"] Oct 31 20:08:17.897310 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:17.897287 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6"] Oct 31 20:08:18.365678 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:18.361211 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/simulator-w1-8b083a-kserve-699c484b58-5v9v6" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 20:08:19.753594 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:19.753561 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" path="/var/lib/kubelet/pods/0cfae09f-0cde-4452-9ac1-babca8bb72a6/volumes" Oct 31 20:08:19.754006 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:19.753991 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" path="/var/lib/kubelet/pods/d01fa9ef-252a-417c-b6d8-27375c4e1b90/volumes" Oct 31 20:08:21.798520 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:21.798478 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" containerID="cri-o://88b8a4e0c49d746c34f4a5b5e669d9d53dad08ec4a8c418926e7dfba0e037f0a" gracePeriod=45 Oct 31 20:08:21.821303 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:21.821273 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" containerID="cri-o://5688f4d0ce672189a34da2747a62fe8eab44fed2b5d1f7df7618e5b9ff1270d7" gracePeriod=45 Oct 31 20:08:21.849419 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:21.849386 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" containerID="cri-o://2d20e17670a21aba5b9495e732fc2ad73f889bf5aa3342087a26713fd657e979" gracePeriod=45 Oct 31 20:08:22.102335 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.102307 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 20:08:22.133731 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.133116 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 20:08:22.163259 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.163225 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-dshm\") pod \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\" (UID: \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\") " Oct 31 20:08:22.163499 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.163290 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/423e19ac-6ae0-4240-9a04-17068d61ebd7-tls-certs\" (UniqueName: \"kubernetes.io/secret/423e19ac-6ae0-4240-9a04-17068d61ebd7-tls-certs\") pod \"423e19ac-6ae0-4240-9a04-17068d61ebd7\" (UID: \"423e19ac-6ae0-4240-9a04-17068d61ebd7\") " Oct 31 20:08:22.163499 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.163326 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-home\" (UniqueName: \"kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-home\") pod \"423e19ac-6ae0-4240-9a04-17068d61ebd7\" (UID: \"423e19ac-6ae0-4240-9a04-17068d61ebd7\") " Oct 31 20:08:22.163499 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.163365 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-tmp-dir\") pod \"423e19ac-6ae0-4240-9a04-17068d61ebd7\" (UID: \"423e19ac-6ae0-4240-9a04-17068d61ebd7\") " Oct 31 20:08:22.163499 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.163433 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bfbef1c-402f-4a21-a378-e4906ec55c4c-kube-api-access-qlh9z\" (UniqueName: \"kubernetes.io/projected/1bfbef1c-402f-4a21-a378-e4906ec55c4c-kube-api-access-qlh9z\") pod \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\" (UID: \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\") " Oct 31 20:08:22.163499 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.163468 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-home\" (UniqueName: \"kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-home\") pod \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\" (UID: \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\") " Oct 31 20:08:22.164045 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.163536 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-model-cache\") pod \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\" (UID: \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\") " Oct 31 20:08:22.164045 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.163585 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-tmp-dir\") pod \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\" (UID: \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\") " Oct 31 20:08:22.164045 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.163619 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/423e19ac-6ae0-4240-9a04-17068d61ebd7-kube-api-access-rhdh2\" (UniqueName: \"kubernetes.io/projected/423e19ac-6ae0-4240-9a04-17068d61ebd7-kube-api-access-rhdh2\") pod \"423e19ac-6ae0-4240-9a04-17068d61ebd7\" (UID: \"423e19ac-6ae0-4240-9a04-17068d61ebd7\") " Oct 31 20:08:22.164045 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.163668 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-model-cache\") pod \"423e19ac-6ae0-4240-9a04-17068d61ebd7\" (UID: \"423e19ac-6ae0-4240-9a04-17068d61ebd7\") " Oct 31 20:08:22.164045 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.163718 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-dshm\") pod \"423e19ac-6ae0-4240-9a04-17068d61ebd7\" (UID: \"423e19ac-6ae0-4240-9a04-17068d61ebd7\") " Oct 31 20:08:22.164373 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.164170 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-tmp-dir" pod "423e19ac-6ae0-4240-9a04-17068d61ebd7" (UID: "423e19ac-6ae0-4240-9a04-17068d61ebd7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:22.166160 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.164736 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-model-cache" pod "1bfbef1c-402f-4a21-a378-e4906ec55c4c" (UID: "1bfbef1c-402f-4a21-a378-e4906ec55c4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:22.166160 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.165475 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-model-cache" pod "423e19ac-6ae0-4240-9a04-17068d61ebd7" (UID: "423e19ac-6ae0-4240-9a04-17068d61ebd7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:22.166160 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.165484 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-home" pod "1bfbef1c-402f-4a21-a378-e4906ec55c4c" (UID: "1bfbef1c-402f-4a21-a378-e4906ec55c4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:22.166160 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.165688 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-home" pod "423e19ac-6ae0-4240-9a04-17068d61ebd7" (UID: "423e19ac-6ae0-4240-9a04-17068d61ebd7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:22.166160 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.165772 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-tmp-dir" pod "1bfbef1c-402f-4a21-a378-e4906ec55c4c" (UID: "1bfbef1c-402f-4a21-a378-e4906ec55c4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:22.166882 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.166856 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-dshm" pod "1bfbef1c-402f-4a21-a378-e4906ec55c4c" (UID: "1bfbef1c-402f-4a21-a378-e4906ec55c4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:22.167702 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.167675 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-dshm" pod "423e19ac-6ae0-4240-9a04-17068d61ebd7" (UID: "423e19ac-6ae0-4240-9a04-17068d61ebd7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:22.167821 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.167738 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/423e19ac-6ae0-4240-9a04-17068d61ebd7-tls-certs" pod "423e19ac-6ae0-4240-9a04-17068d61ebd7" (UID: "423e19ac-6ae0-4240-9a04-17068d61ebd7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:08:22.169273 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.169241 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423e19ac-6ae0-4240-9a04-17068d61ebd7-kube-api-access-rhdh2" pod "423e19ac-6ae0-4240-9a04-17068d61ebd7" (UID: "423e19ac-6ae0-4240-9a04-17068d61ebd7"). InnerVolumeSpecName "kube-api-access-rhdh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:08:22.169814 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.169788 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bfbef1c-402f-4a21-a378-e4906ec55c4c-kube-api-access-qlh9z" pod "1bfbef1c-402f-4a21-a378-e4906ec55c4c" (UID: "1bfbef1c-402f-4a21-a378-e4906ec55c4c"). InnerVolumeSpecName "kube-api-access-qlh9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:08:22.177164 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.177139 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 20:08:22.264565 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.264533 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bfbef1c-402f-4a21-a378-e4906ec55c4c-tls-certs\" (UniqueName: \"kubernetes.io/secret/1bfbef1c-402f-4a21-a378-e4906ec55c4c-tls-certs\") pod \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\" (UID: \"1bfbef1c-402f-4a21-a378-e4906ec55c4c\") " Oct 31 20:08:22.264725 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.264576 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97560889-c009-4292-968a-58758bd2266b-tls-certs\" (UniqueName: \"kubernetes.io/secret/97560889-c009-4292-968a-58758bd2266b-tls-certs\") pod \"97560889-c009-4292-968a-58758bd2266b\" (UID: \"97560889-c009-4292-968a-58758bd2266b\") " Oct 31 20:08:22.264725 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.264607 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-home\" (UniqueName: \"kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-home\") pod \"97560889-c009-4292-968a-58758bd2266b\" (UID: \"97560889-c009-4292-968a-58758bd2266b\") " Oct 31 20:08:22.264725 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.264642 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97560889-c009-4292-968a-58758bd2266b-kube-api-access-v4kdh\" (UniqueName: \"kubernetes.io/projected/97560889-c009-4292-968a-58758bd2266b-kube-api-access-v4kdh\") pod \"97560889-c009-4292-968a-58758bd2266b\" (UID: \"97560889-c009-4292-968a-58758bd2266b\") " Oct 31 20:08:22.264725 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.264665 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-model-cache\") pod \"97560889-c009-4292-968a-58758bd2266b\" (UID: \"97560889-c009-4292-968a-58758bd2266b\") " Oct 31 20:08:22.265044 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.264746 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-tmp-dir\") pod \"97560889-c009-4292-968a-58758bd2266b\" (UID: \"97560889-c009-4292-968a-58758bd2266b\") " Oct 31 20:08:22.265044 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.264991 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-home" pod "97560889-c009-4292-968a-58758bd2266b" (UID: "97560889-c009-4292-968a-58758bd2266b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:22.265174 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.265108 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-model-cache" pod "97560889-c009-4292-968a-58758bd2266b" (UID: "97560889-c009-4292-968a-58758bd2266b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:22.265245 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.265169 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-dshm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:22.265245 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.265189 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-dshm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:22.265245 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.265201 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-tmp-dir" pod "97560889-c009-4292-968a-58758bd2266b" (UID: "97560889-c009-4292-968a-58758bd2266b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:22.265245 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.265206 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-home\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:22.265245 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.265230 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/423e19ac-6ae0-4240-9a04-17068d61ebd7-tls-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:22.265245 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.265245 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-home\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:22.265514 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.265259 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-tmp-dir\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:22.265514 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.265273 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlh9z\" (UniqueName: \"kubernetes.io/projected/1bfbef1c-402f-4a21-a378-e4906ec55c4c-kube-api-access-qlh9z\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:22.265514 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.265288 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-home\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:22.265514 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.265297 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-model-cache\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:22.265514 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.265307 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bfbef1c-402f-4a21-a378-e4906ec55c4c-tmp-dir\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:22.265514 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.265325 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhdh2\" (UniqueName: \"kubernetes.io/projected/423e19ac-6ae0-4240-9a04-17068d61ebd7-kube-api-access-rhdh2\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:22.265514 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.265343 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/423e19ac-6ae0-4240-9a04-17068d61ebd7-model-cache\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:22.266906 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.266884 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97560889-c009-4292-968a-58758bd2266b-kube-api-access-v4kdh" pod "97560889-c009-4292-968a-58758bd2266b" (UID: "97560889-c009-4292-968a-58758bd2266b"). InnerVolumeSpecName "kube-api-access-v4kdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:08:22.267082 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.267062 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bfbef1c-402f-4a21-a378-e4906ec55c4c-tls-certs" pod "1bfbef1c-402f-4a21-a378-e4906ec55c4c" (UID: "1bfbef1c-402f-4a21-a378-e4906ec55c4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:08:22.267357 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.267343 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97560889-c009-4292-968a-58758bd2266b-tls-certs" pod "97560889-c009-4292-968a-58758bd2266b" (UID: "97560889-c009-4292-968a-58758bd2266b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:08:22.365796 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.365757 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-dshm\") pod \"97560889-c009-4292-968a-58758bd2266b\" (UID: \"97560889-c009-4292-968a-58758bd2266b\") " Oct 31 20:08:22.366035 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.366009 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-tmp-dir\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:22.366035 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.366026 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bfbef1c-402f-4a21-a378-e4906ec55c4c-tls-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:22.366035 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.366035 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97560889-c009-4292-968a-58758bd2266b-tls-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:22.366147 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.366045 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4kdh\" (UniqueName: \"kubernetes.io/projected/97560889-c009-4292-968a-58758bd2266b-kube-api-access-v4kdh\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:22.366147 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.366054 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-model-cache\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:22.367818 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.367795 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-dshm" pod "97560889-c009-4292-968a-58758bd2266b" (UID: "97560889-c009-4292-968a-58758bd2266b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:22.466632 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.466589 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97560889-c009-4292-968a-58758bd2266b-dshm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:22.845739 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.845704 4910 generic.go:356] "Generic (PLEG): container finished" podID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerID="2d20e17670a21aba5b9495e732fc2ad73f889bf5aa3342087a26713fd657e979" exitCode=0 Oct 31 20:08:22.846234 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.845750 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" Oct 31 20:08:22.846234 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.845782 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" event={"ID":"423e19ac-6ae0-4240-9a04-17068d61ebd7","Type":"ContainerDied","Data":"2d20e17670a21aba5b9495e732fc2ad73f889bf5aa3342087a26713fd657e979"} Oct 31 20:08:22.846234 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.845820 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb" event={"ID":"423e19ac-6ae0-4240-9a04-17068d61ebd7","Type":"ContainerDied","Data":"dd63cc1040f37f3b138c3d78c3d56fb0d4b956fc25abef5a8664d2d617f68e6b"} Oct 31 20:08:22.846234 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.845841 4910 scope.go:122] "RemoveContainer" containerID="2d20e17670a21aba5b9495e732fc2ad73f889bf5aa3342087a26713fd657e979" Oct 31 20:08:22.847438 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.847413 4910 generic.go:356] "Generic (PLEG): container finished" podID="97560889-c009-4292-968a-58758bd2266b" containerID="88b8a4e0c49d746c34f4a5b5e669d9d53dad08ec4a8c418926e7dfba0e037f0a" exitCode=0 Oct 31 20:08:22.847573 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.847447 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" event={"ID":"97560889-c009-4292-968a-58758bd2266b","Type":"ContainerDied","Data":"88b8a4e0c49d746c34f4a5b5e669d9d53dad08ec4a8c418926e7dfba0e037f0a"} Oct 31 20:08:22.847573 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.847477 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" event={"ID":"97560889-c009-4292-968a-58758bd2266b","Type":"ContainerDied","Data":"ceca75d4a3d0dc3a6cb787d7cf131cd72c88296b4b1a9f739fc73d6d4f3f53dd"} Oct 31 20:08:22.847573 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.847475 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n" Oct 31 20:08:22.849113 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.849093 4910 generic.go:356] "Generic (PLEG): container finished" podID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerID="5688f4d0ce672189a34da2747a62fe8eab44fed2b5d1f7df7618e5b9ff1270d7" exitCode=0 Oct 31 20:08:22.849225 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.849140 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" Oct 31 20:08:22.849225 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.849129 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" event={"ID":"1bfbef1c-402f-4a21-a378-e4906ec55c4c","Type":"ContainerDied","Data":"5688f4d0ce672189a34da2747a62fe8eab44fed2b5d1f7df7618e5b9ff1270d7"} Oct 31 20:08:22.849225 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.849191 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws" event={"ID":"1bfbef1c-402f-4a21-a378-e4906ec55c4c","Type":"ContainerDied","Data":"98c18d8441673a9414552a36a894975d8707ad4770468e314a535dee7d8a09a4"} Oct 31 20:08:22.858256 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.858227 4910 scope.go:122] "RemoveContainer" containerID="2d20e17670a21aba5b9495e732fc2ad73f889bf5aa3342087a26713fd657e979" Oct 31 20:08:22.858589 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:08:22.858513 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d20e17670a21aba5b9495e732fc2ad73f889bf5aa3342087a26713fd657e979\": container with ID starting with 2d20e17670a21aba5b9495e732fc2ad73f889bf5aa3342087a26713fd657e979 not found: ID does not exist" containerID="2d20e17670a21aba5b9495e732fc2ad73f889bf5aa3342087a26713fd657e979" Oct 31 20:08:22.858589 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.858555 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d20e17670a21aba5b9495e732fc2ad73f889bf5aa3342087a26713fd657e979"} err="failed to get container status \"2d20e17670a21aba5b9495e732fc2ad73f889bf5aa3342087a26713fd657e979\": rpc error: code = NotFound desc = could not find container \"2d20e17670a21aba5b9495e732fc2ad73f889bf5aa3342087a26713fd657e979\": container with ID starting with 2d20e17670a21aba5b9495e732fc2ad73f889bf5aa3342087a26713fd657e979 not found: ID does not exist" Oct 31 20:08:22.858589 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.858578 4910 scope.go:122] "RemoveContainer" containerID="88b8a4e0c49d746c34f4a5b5e669d9d53dad08ec4a8c418926e7dfba0e037f0a" Oct 31 20:08:22.867460 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.867442 4910 scope.go:122] "RemoveContainer" containerID="88b8a4e0c49d746c34f4a5b5e669d9d53dad08ec4a8c418926e7dfba0e037f0a" Oct 31 20:08:22.867701 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:08:22.867683 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88b8a4e0c49d746c34f4a5b5e669d9d53dad08ec4a8c418926e7dfba0e037f0a\": container with ID starting with 88b8a4e0c49d746c34f4a5b5e669d9d53dad08ec4a8c418926e7dfba0e037f0a not found: ID does not exist" containerID="88b8a4e0c49d746c34f4a5b5e669d9d53dad08ec4a8c418926e7dfba0e037f0a" Oct 31 20:08:22.867747 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.867708 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88b8a4e0c49d746c34f4a5b5e669d9d53dad08ec4a8c418926e7dfba0e037f0a"} err="failed to get container status \"88b8a4e0c49d746c34f4a5b5e669d9d53dad08ec4a8c418926e7dfba0e037f0a\": rpc error: code = NotFound desc = could not find container \"88b8a4e0c49d746c34f4a5b5e669d9d53dad08ec4a8c418926e7dfba0e037f0a\": container with ID starting with 88b8a4e0c49d746c34f4a5b5e669d9d53dad08ec4a8c418926e7dfba0e037f0a not found: ID does not exist" Oct 31 20:08:22.867747 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.867723 4910 scope.go:122] "RemoveContainer" containerID="5688f4d0ce672189a34da2747a62fe8eab44fed2b5d1f7df7618e5b9ff1270d7" Oct 31 20:08:22.877705 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.877682 4910 scope.go:122] "RemoveContainer" containerID="5688f4d0ce672189a34da2747a62fe8eab44fed2b5d1f7df7618e5b9ff1270d7" Oct 31 20:08:22.877846 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.877825 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb"] Oct 31 20:08:22.878113 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:08:22.878051 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5688f4d0ce672189a34da2747a62fe8eab44fed2b5d1f7df7618e5b9ff1270d7\": container with ID starting with 5688f4d0ce672189a34da2747a62fe8eab44fed2b5d1f7df7618e5b9ff1270d7 not found: ID does not exist" containerID="5688f4d0ce672189a34da2747a62fe8eab44fed2b5d1f7df7618e5b9ff1270d7" Oct 31 20:08:22.878113 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.878082 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5688f4d0ce672189a34da2747a62fe8eab44fed2b5d1f7df7618e5b9ff1270d7"} err="failed to get container status \"5688f4d0ce672189a34da2747a62fe8eab44fed2b5d1f7df7618e5b9ff1270d7\": rpc error: code = NotFound desc = could not find container \"5688f4d0ce672189a34da2747a62fe8eab44fed2b5d1f7df7618e5b9ff1270d7\": container with ID starting with 5688f4d0ce672189a34da2747a62fe8eab44fed2b5d1f7df7618e5b9ff1270d7 not found: ID does not exist" Oct 31 20:08:22.882358 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.882337 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-23814d/simulator-w3-23814d-kserve-b47579674-t5hnb"] Oct 31 20:08:22.898094 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.898073 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws"] Oct 31 20:08:22.903267 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.903244 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-23814d/distinct-w3-23814d-kserve-8674c8fb86-2frws"] Oct 31 20:08:22.914648 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.914627 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n"] Oct 31 20:08:22.919531 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:22.919510 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-23814d/distinct-2-w3-23814d-kserve-7588856589-mw99n"] Oct 31 20:08:23.753289 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:23.753208 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" path="/var/lib/kubelet/pods/1bfbef1c-402f-4a21-a378-e4906ec55c4c/volumes" Oct 31 20:08:23.753635 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:23.753621 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" path="/var/lib/kubelet/pods/423e19ac-6ae0-4240-9a04-17068d61ebd7/volumes" Oct 31 20:08:23.754002 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:23.753990 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97560889-c009-4292-968a-58758bd2266b" path="/var/lib/kubelet/pods/97560889-c009-4292-968a-58758bd2266b/volumes" Oct 31 20:08:25.032349 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.032312 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9d24fb1-f4g8k"] Oct 31 20:08:25.040051 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.040022 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9d24fb1-f4g8k"] Oct 31 20:08:25.415317 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.415263 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt"] Oct 31 20:08:25.415687 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.415650 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt" podUID="20862e55-868f-40e8-af36-4f3e30cf9361" containerName="maas-api" containerID="cri-o://3058a471f865d33201bb560a8b7a3031ebdee9a36c7f050af48f189b7992c3d1" gracePeriod=30 Oct 31 20:08:25.428937 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.422891 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz" podUID="9a4949e1-4900-4ce8-9734-ce6e73bef410" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:08:25.694662 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.694638 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt" Oct 31 20:08:25.754245 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.754214 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f768640-93b1-4b5a-bbf8-713c315757bb" path="/var/lib/kubelet/pods/6f768640-93b1-4b5a-bbf8-713c315757bb/volumes" Oct 31 20:08:25.790859 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.790826 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20862e55-868f-40e8-af36-4f3e30cf9361-metrics-certs\" (UniqueName: \"kubernetes.io/secret/20862e55-868f-40e8-af36-4f3e30cf9361-metrics-certs\") pod \"20862e55-868f-40e8-af36-4f3e30cf9361\" (UID: \"20862e55-868f-40e8-af36-4f3e30cf9361\") " Oct 31 20:08:25.791108 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.790868 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20862e55-868f-40e8-af36-4f3e30cf9361-kube-api-access-rj8lk\" (UniqueName: \"kubernetes.io/projected/20862e55-868f-40e8-af36-4f3e30cf9361-kube-api-access-rj8lk\") pod \"20862e55-868f-40e8-af36-4f3e30cf9361\" (UID: \"20862e55-868f-40e8-af36-4f3e30cf9361\") " Oct 31 20:08:25.791108 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.790900 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20862e55-868f-40e8-af36-4f3e30cf9361-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20862e55-868f-40e8-af36-4f3e30cf9361-openshift-service-ca\") pod \"20862e55-868f-40e8-af36-4f3e30cf9361\" (UID: \"20862e55-868f-40e8-af36-4f3e30cf9361\") " Oct 31 20:08:25.791108 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.790984 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20862e55-868f-40e8-af36-4f3e30cf9361-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20862e55-868f-40e8-af36-4f3e30cf9361-maas-api-tls\") pod \"20862e55-868f-40e8-af36-4f3e30cf9361\" (UID: \"20862e55-868f-40e8-af36-4f3e30cf9361\") " Oct 31 20:08:25.791348 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.791317 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20862e55-868f-40e8-af36-4f3e30cf9361-openshift-service-ca" pod "20862e55-868f-40e8-af36-4f3e30cf9361" (UID: "20862e55-868f-40e8-af36-4f3e30cf9361"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:08:25.793168 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.793136 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20862e55-868f-40e8-af36-4f3e30cf9361-metrics-certs" pod "20862e55-868f-40e8-af36-4f3e30cf9361" (UID: "20862e55-868f-40e8-af36-4f3e30cf9361"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:08:25.793271 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.793196 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20862e55-868f-40e8-af36-4f3e30cf9361-kube-api-access-rj8lk" pod "20862e55-868f-40e8-af36-4f3e30cf9361" (UID: "20862e55-868f-40e8-af36-4f3e30cf9361"). InnerVolumeSpecName "kube-api-access-rj8lk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:08:25.793312 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.793247 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20862e55-868f-40e8-af36-4f3e30cf9361-maas-api-tls" pod "20862e55-868f-40e8-af36-4f3e30cf9361" (UID: "20862e55-868f-40e8-af36-4f3e30cf9361"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:08:25.881010 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.880813 4910 generic.go:356] "Generic (PLEG): container finished" podID="20862e55-868f-40e8-af36-4f3e30cf9361" containerID="3058a471f865d33201bb560a8b7a3031ebdee9a36c7f050af48f189b7992c3d1" exitCode=0 Oct 31 20:08:25.881010 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.880963 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt" event={"ID":"20862e55-868f-40e8-af36-4f3e30cf9361","Type":"ContainerDied","Data":"3058a471f865d33201bb560a8b7a3031ebdee9a36c7f050af48f189b7992c3d1"} Oct 31 20:08:25.881341 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.881310 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt" event={"ID":"20862e55-868f-40e8-af36-4f3e30cf9361","Type":"ContainerDied","Data":"8eeab951271e1db46e199f7bdff71cff017e037912fc399c5ee6d9a390068cde"} Oct 31 20:08:25.881489 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.881362 4910 scope.go:122] "RemoveContainer" containerID="3058a471f865d33201bb560a8b7a3031ebdee9a36c7f050af48f189b7992c3d1" Oct 31 20:08:25.881580 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.881521 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt" Oct 31 20:08:25.891939 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.891882 4910 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20862e55-868f-40e8-af36-4f3e30cf9361-metrics-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:25.893089 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.892184 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj8lk\" (UniqueName: \"kubernetes.io/projected/20862e55-868f-40e8-af36-4f3e30cf9361-kube-api-access-rj8lk\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:25.893089 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.892229 4910 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20862e55-868f-40e8-af36-4f3e30cf9361-openshift-service-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:25.893089 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.892248 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20862e55-868f-40e8-af36-4f3e30cf9361-maas-api-tls\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:25.906120 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.906096 4910 scope.go:122] "RemoveContainer" containerID="3058a471f865d33201bb560a8b7a3031ebdee9a36c7f050af48f189b7992c3d1" Oct 31 20:08:25.906458 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:08:25.906430 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3058a471f865d33201bb560a8b7a3031ebdee9a36c7f050af48f189b7992c3d1\": container with ID starting with 3058a471f865d33201bb560a8b7a3031ebdee9a36c7f050af48f189b7992c3d1 not found: ID does not exist" containerID="3058a471f865d33201bb560a8b7a3031ebdee9a36c7f050af48f189b7992c3d1" Oct 31 20:08:25.906509 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.906468 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3058a471f865d33201bb560a8b7a3031ebdee9a36c7f050af48f189b7992c3d1"} err="failed to get container status \"3058a471f865d33201bb560a8b7a3031ebdee9a36c7f050af48f189b7992c3d1\": rpc error: code = NotFound desc = could not find container \"3058a471f865d33201bb560a8b7a3031ebdee9a36c7f050af48f189b7992c3d1\": container with ID starting with 3058a471f865d33201bb560a8b7a3031ebdee9a36c7f050af48f189b7992c3d1 not found: ID does not exist" Oct 31 20:08:25.939778 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.939691 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt"] Oct 31 20:08:25.949251 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:25.949218 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-23814d-b77d479bb-b8znt"] Oct 31 20:08:27.753782 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:27.753747 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20862e55-868f-40e8-af36-4f3e30cf9361" path="/var/lib/kubelet/pods/20862e55-868f-40e8-af36-4f3e30cf9361/volumes" Oct 31 20:08:27.934139 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:27.934098 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4"] Oct 31 20:08:27.934353 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:27.934326 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" podUID="cb65a183-bd28-4a21-8895-82b7a39b4693" containerName="istio-proxy" containerID="cri-o://0604db300e013ac905efb7c7adb034c94443496323246d49de6854ffd25fb06f" gracePeriod=30 Oct 31 20:08:32.203118 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:32.203083 4910 scope.go:122] "RemoveContainer" containerID="120ac1858acb982dcd1e9d0eab1792f30569089cd0ce5475de6ebe1c954a6033" Oct 31 20:08:33.184542 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.184513 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 20:08:33.259454 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.259372 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb65a183-bd28-4a21-8895-82b7a39b4693-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb65a183-bd28-4a21-8895-82b7a39b4693-istiod-ca-cert\") pod \"cb65a183-bd28-4a21-8895-82b7a39b4693\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " Oct 31 20:08:33.259454 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.259446 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-token\" (UniqueName: \"kubernetes.io/projected/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-token\") pod \"cb65a183-bd28-4a21-8895-82b7a39b4693\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " Oct 31 20:08:33.260024 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.259474 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-credential-socket\") pod \"cb65a183-bd28-4a21-8895-82b7a39b4693\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " Oct 31 20:08:33.260024 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.259495 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-workload-certs\") pod \"cb65a183-bd28-4a21-8895-82b7a39b4693\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " Oct 31 20:08:33.260024 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.259528 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-data\") pod \"cb65a183-bd28-4a21-8895-82b7a39b4693\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " Oct 31 20:08:33.260024 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.259578 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-workload-socket\") pod \"cb65a183-bd28-4a21-8895-82b7a39b4693\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " Oct 31 20:08:33.260024 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.259634 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-envoy\") pod \"cb65a183-bd28-4a21-8895-82b7a39b4693\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " Oct 31 20:08:33.260024 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.259672 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb65a183-bd28-4a21-8895-82b7a39b4693-kube-api-access-wrfqm\" (UniqueName: \"kubernetes.io/projected/cb65a183-bd28-4a21-8895-82b7a39b4693-kube-api-access-wrfqm\") pod \"cb65a183-bd28-4a21-8895-82b7a39b4693\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " Oct 31 20:08:33.260024 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.259747 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-podinfo\") pod \"cb65a183-bd28-4a21-8895-82b7a39b4693\" (UID: \"cb65a183-bd28-4a21-8895-82b7a39b4693\") " Oct 31 20:08:33.260024 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.259814 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb65a183-bd28-4a21-8895-82b7a39b4693-istiod-ca-cert" pod "cb65a183-bd28-4a21-8895-82b7a39b4693" (UID: "cb65a183-bd28-4a21-8895-82b7a39b4693"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:08:33.260024 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.259939 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-workload-socket" pod "cb65a183-bd28-4a21-8895-82b7a39b4693" (UID: "cb65a183-bd28-4a21-8895-82b7a39b4693"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:33.260024 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.259904 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-workload-certs" pod "cb65a183-bd28-4a21-8895-82b7a39b4693" (UID: "cb65a183-bd28-4a21-8895-82b7a39b4693"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:33.260623 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.260108 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb65a183-bd28-4a21-8895-82b7a39b4693-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:33.260623 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.260127 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:33.260623 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.260142 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:33.260623 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.260148 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-credential-socket" pod "cb65a183-bd28-4a21-8895-82b7a39b4693" (UID: "cb65a183-bd28-4a21-8895-82b7a39b4693"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:33.260623 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.260384 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-data" pod "cb65a183-bd28-4a21-8895-82b7a39b4693" (UID: "cb65a183-bd28-4a21-8895-82b7a39b4693"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:33.262203 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.262174 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb65a183-bd28-4a21-8895-82b7a39b4693-kube-api-access-wrfqm" pod "cb65a183-bd28-4a21-8895-82b7a39b4693" (UID: "cb65a183-bd28-4a21-8895-82b7a39b4693"). InnerVolumeSpecName "kube-api-access-wrfqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:08:33.262337 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.262210 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-podinfo" pod "cb65a183-bd28-4a21-8895-82b7a39b4693" (UID: "cb65a183-bd28-4a21-8895-82b7a39b4693"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:08:33.262413 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.262385 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-envoy" pod "cb65a183-bd28-4a21-8895-82b7a39b4693" (UID: "cb65a183-bd28-4a21-8895-82b7a39b4693"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:33.262821 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.262799 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-token" pod "cb65a183-bd28-4a21-8895-82b7a39b4693" (UID: "cb65a183-bd28-4a21-8895-82b7a39b4693"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:08:33.361500 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.361454 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:33.361500 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.361489 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:33.361500 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.361499 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:33.361500 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.361508 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:33.361880 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.361521 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrfqm\" (UniqueName: \"kubernetes.io/projected/cb65a183-bd28-4a21-8895-82b7a39b4693-kube-api-access-wrfqm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:33.361880 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.361531 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb65a183-bd28-4a21-8895-82b7a39b4693-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:33.950217 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.950178 4910 generic.go:356] "Generic (PLEG): container finished" podID="cb65a183-bd28-4a21-8895-82b7a39b4693" containerID="0604db300e013ac905efb7c7adb034c94443496323246d49de6854ffd25fb06f" exitCode=0 Oct 31 20:08:33.951157 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.950256 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" Oct 31 20:08:33.951157 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.950271 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" event={"ID":"cb65a183-bd28-4a21-8895-82b7a39b4693","Type":"ContainerDied","Data":"0604db300e013ac905efb7c7adb034c94443496323246d49de6854ffd25fb06f"} Oct 31 20:08:33.951157 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.950337 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4" event={"ID":"cb65a183-bd28-4a21-8895-82b7a39b4693","Type":"ContainerDied","Data":"232453e97007bfbcb8416b1085daedfa3902cf855f54a7626364fa9f3201f04f"} Oct 31 20:08:33.951157 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.950374 4910 scope.go:122] "RemoveContainer" containerID="0604db300e013ac905efb7c7adb034c94443496323246d49de6854ffd25fb06f" Oct 31 20:08:33.992251 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:33.992201 4910 scope.go:122] "RemoveContainer" containerID="5d06458dd11bddf40af64d46b76348c5511f5f103928db85015ae94810300edc" Oct 31 20:08:34.005038 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:34.005003 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4"] Oct 31 20:08:34.009325 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:34.009277 4910 scope.go:122] "RemoveContainer" containerID="0604db300e013ac905efb7c7adb034c94443496323246d49de6854ffd25fb06f" Oct 31 20:08:34.009691 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:08:34.009669 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0604db300e013ac905efb7c7adb034c94443496323246d49de6854ffd25fb06f\": container with ID starting with 0604db300e013ac905efb7c7adb034c94443496323246d49de6854ffd25fb06f not found: ID does not exist" containerID="0604db300e013ac905efb7c7adb034c94443496323246d49de6854ffd25fb06f" Oct 31 20:08:34.009799 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:34.009707 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0604db300e013ac905efb7c7adb034c94443496323246d49de6854ffd25fb06f"} err="failed to get container status \"0604db300e013ac905efb7c7adb034c94443496323246d49de6854ffd25fb06f\": rpc error: code = NotFound desc = could not find container \"0604db300e013ac905efb7c7adb034c94443496323246d49de6854ffd25fb06f\": container with ID starting with 0604db300e013ac905efb7c7adb034c94443496323246d49de6854ffd25fb06f not found: ID does not exist" Oct 31 20:08:34.009799 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:34.009735 4910 scope.go:122] "RemoveContainer" containerID="5d06458dd11bddf40af64d46b76348c5511f5f103928db85015ae94810300edc" Oct 31 20:08:34.010197 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:08:34.010170 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d06458dd11bddf40af64d46b76348c5511f5f103928db85015ae94810300edc\": container with ID starting with 5d06458dd11bddf40af64d46b76348c5511f5f103928db85015ae94810300edc not found: ID does not exist" containerID="5d06458dd11bddf40af64d46b76348c5511f5f103928db85015ae94810300edc" Oct 31 20:08:34.010284 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:34.010207 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d06458dd11bddf40af64d46b76348c5511f5f103928db85015ae94810300edc"} err="failed to get container status \"5d06458dd11bddf40af64d46b76348c5511f5f103928db85015ae94810300edc\": rpc error: code = NotFound desc = could not find container \"5d06458dd11bddf40af64d46b76348c5511f5f103928db85015ae94810300edc\": container with ID starting with 5d06458dd11bddf40af64d46b76348c5511f5f103928db85015ae94810300edc not found: ID does not exist" Oct 31 20:08:34.017152 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:34.017092 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr"] Oct 31 20:08:34.017302 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:34.017284 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" podUID="c3653569-1618-48e9-803d-5cb94a356041" containerName="istio-proxy" containerID="cri-o://4e1129c1e6cf16ad431c16d9f3a49e3559a535c1eba14676af9fe0b87196a128" gracePeriod=30 Oct 31 20:08:34.023129 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:34.023102 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-j7bc4"] Oct 31 20:08:35.419948 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:35.419890 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz" podUID="9a4949e1-4900-4ce8-9734-ce6e73bef410" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 20:08:35.754049 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:35.753946 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb65a183-bd28-4a21-8895-82b7a39b4693" path="/var/lib/kubelet/pods/cb65a183-bd28-4a21-8895-82b7a39b4693/volumes" Oct 31 20:08:37.498511 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:37.498485 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz" Oct 31 20:08:37.630751 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:37.630711 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a4949e1-4900-4ce8-9734-ce6e73bef410-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a4949e1-4900-4ce8-9734-ce6e73bef410-plugins-config-volume\") pod \"9a4949e1-4900-4ce8-9734-ce6e73bef410\" (UID: \"9a4949e1-4900-4ce8-9734-ce6e73bef410\") " Oct 31 20:08:37.630975 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:37.630776 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a4949e1-4900-4ce8-9734-ce6e73bef410-kube-api-access-ssdsx\" (UniqueName: \"kubernetes.io/projected/9a4949e1-4900-4ce8-9734-ce6e73bef410-kube-api-access-ssdsx\") pod \"9a4949e1-4900-4ce8-9734-ce6e73bef410\" (UID: \"9a4949e1-4900-4ce8-9734-ce6e73bef410\") " Oct 31 20:08:37.631246 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:37.631204 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a4949e1-4900-4ce8-9734-ce6e73bef410-plugins-config-volume" pod "9a4949e1-4900-4ce8-9734-ce6e73bef410" (UID: "9a4949e1-4900-4ce8-9734-ce6e73bef410"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:08:37.632875 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:37.632849 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a4949e1-4900-4ce8-9734-ce6e73bef410-kube-api-access-ssdsx" pod "9a4949e1-4900-4ce8-9734-ce6e73bef410" (UID: "9a4949e1-4900-4ce8-9734-ce6e73bef410"). InnerVolumeSpecName "kube-api-access-ssdsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:08:37.731607 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:37.731565 4910 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a4949e1-4900-4ce8-9734-ce6e73bef410-plugins-config-volume\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:37.731607 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:37.731610 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssdsx\" (UniqueName: \"kubernetes.io/projected/9a4949e1-4900-4ce8-9734-ce6e73bef410-kube-api-access-ssdsx\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:37.975814 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:37.975724 4910 generic.go:356] "Generic (PLEG): container finished" podID="9a4949e1-4900-4ce8-9734-ce6e73bef410" containerID="467a63236c40f6886d5f0c36f5079efa32cbc036ae0bdeefb18b4c0e4007267b" exitCode=0 Oct 31 20:08:37.975814 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:37.975773 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz" Oct 31 20:08:37.976034 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:37.975815 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz" event={"ID":"9a4949e1-4900-4ce8-9734-ce6e73bef410","Type":"ContainerDied","Data":"467a63236c40f6886d5f0c36f5079efa32cbc036ae0bdeefb18b4c0e4007267b"} Oct 31 20:08:37.976034 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:37.975873 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz" event={"ID":"9a4949e1-4900-4ce8-9734-ce6e73bef410","Type":"ContainerDied","Data":"fea6cee7f9ca11971274a83c0b6743f4bb2bcf4175ef41eabb0cce56ddde57a6"} Oct 31 20:08:37.976034 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:37.975904 4910 scope.go:122] "RemoveContainer" containerID="467a63236c40f6886d5f0c36f5079efa32cbc036ae0bdeefb18b4c0e4007267b" Oct 31 20:08:37.985337 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:37.985319 4910 scope.go:122] "RemoveContainer" containerID="467a63236c40f6886d5f0c36f5079efa32cbc036ae0bdeefb18b4c0e4007267b" Oct 31 20:08:37.985616 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:08:37.985590 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"467a63236c40f6886d5f0c36f5079efa32cbc036ae0bdeefb18b4c0e4007267b\": container with ID starting with 467a63236c40f6886d5f0c36f5079efa32cbc036ae0bdeefb18b4c0e4007267b not found: ID does not exist" containerID="467a63236c40f6886d5f0c36f5079efa32cbc036ae0bdeefb18b4c0e4007267b" Oct 31 20:08:37.985679 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:37.985623 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"467a63236c40f6886d5f0c36f5079efa32cbc036ae0bdeefb18b4c0e4007267b"} err="failed to get container status \"467a63236c40f6886d5f0c36f5079efa32cbc036ae0bdeefb18b4c0e4007267b\": rpc error: code = NotFound desc = could not find container \"467a63236c40f6886d5f0c36f5079efa32cbc036ae0bdeefb18b4c0e4007267b\": container with ID starting with 467a63236c40f6886d5f0c36f5079efa32cbc036ae0bdeefb18b4c0e4007267b not found: ID does not exist" Oct 31 20:08:38.008618 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:38.008587 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz"] Oct 31 20:08:38.021264 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:38.021234 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-23814d-75cfff9b75-qwtzz"] Oct 31 20:08:39.294279 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.294212 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 20:08:39.446304 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.446270 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-istio-envoy\") pod \"c3653569-1618-48e9-803d-5cb94a356041\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " Oct 31 20:08:39.446526 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.446322 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-credential-socket\") pod \"c3653569-1618-48e9-803d-5cb94a356041\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " Oct 31 20:08:39.446526 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.446355 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3653569-1618-48e9-803d-5cb94a356041-istio-token\" (UniqueName: \"kubernetes.io/projected/c3653569-1618-48e9-803d-5cb94a356041-istio-token\") pod \"c3653569-1618-48e9-803d-5cb94a356041\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " Oct 31 20:08:39.446526 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.446402 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-istio-data\") pod \"c3653569-1618-48e9-803d-5cb94a356041\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " Oct 31 20:08:39.446526 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.446450 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3653569-1618-48e9-803d-5cb94a356041-kube-api-access-2t5kd\" (UniqueName: \"kubernetes.io/projected/c3653569-1618-48e9-803d-5cb94a356041-kube-api-access-2t5kd\") pod \"c3653569-1618-48e9-803d-5cb94a356041\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " Oct 31 20:08:39.446964 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.446537 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3653569-1618-48e9-803d-5cb94a356041-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3653569-1618-48e9-803d-5cb94a356041-istio-podinfo\") pod \"c3653569-1618-48e9-803d-5cb94a356041\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " Oct 31 20:08:39.446964 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.446572 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-workload-socket\") pod \"c3653569-1618-48e9-803d-5cb94a356041\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " Oct 31 20:08:39.446964 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.446604 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3653569-1618-48e9-803d-5cb94a356041-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3653569-1618-48e9-803d-5cb94a356041-istiod-ca-cert\") pod \"c3653569-1618-48e9-803d-5cb94a356041\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " Oct 31 20:08:39.446964 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.446648 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-workload-certs\") pod \"c3653569-1618-48e9-803d-5cb94a356041\" (UID: \"c3653569-1618-48e9-803d-5cb94a356041\") " Oct 31 20:08:39.447592 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.447185 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-workload-certs" pod "c3653569-1618-48e9-803d-5cb94a356041" (UID: "c3653569-1618-48e9-803d-5cb94a356041"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:39.447592 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.447341 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-workload-socket" pod "c3653569-1618-48e9-803d-5cb94a356041" (UID: "c3653569-1618-48e9-803d-5cb94a356041"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:39.447592 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.447434 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-istio-data" pod "c3653569-1618-48e9-803d-5cb94a356041" (UID: "c3653569-1618-48e9-803d-5cb94a356041"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:39.447592 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.447547 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3653569-1618-48e9-803d-5cb94a356041-istiod-ca-cert" pod "c3653569-1618-48e9-803d-5cb94a356041" (UID: "c3653569-1618-48e9-803d-5cb94a356041"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:08:39.447592 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.447569 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-credential-socket" pod "c3653569-1618-48e9-803d-5cb94a356041" (UID: "c3653569-1618-48e9-803d-5cb94a356041"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:39.451646 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.450731 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-istio-envoy" pod "c3653569-1618-48e9-803d-5cb94a356041" (UID: "c3653569-1618-48e9-803d-5cb94a356041"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:08:39.451646 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.451455 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3653569-1618-48e9-803d-5cb94a356041-istio-token" pod "c3653569-1618-48e9-803d-5cb94a356041" (UID: "c3653569-1618-48e9-803d-5cb94a356041"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:08:39.451646 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.451548 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3653569-1618-48e9-803d-5cb94a356041-istio-podinfo" pod "c3653569-1618-48e9-803d-5cb94a356041" (UID: "c3653569-1618-48e9-803d-5cb94a356041"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:08:39.451646 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.451627 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3653569-1618-48e9-803d-5cb94a356041-kube-api-access-2t5kd" pod "c3653569-1618-48e9-803d-5cb94a356041" (UID: "c3653569-1618-48e9-803d-5cb94a356041"). InnerVolumeSpecName "kube-api-access-2t5kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:08:39.547966 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.547841 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:39.547966 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.547888 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t5kd\" (UniqueName: \"kubernetes.io/projected/c3653569-1618-48e9-803d-5cb94a356041-kube-api-access-2t5kd\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:39.547966 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.547905 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3653569-1618-48e9-803d-5cb94a356041-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:39.547966 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.547949 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:39.547966 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.547964 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3653569-1618-48e9-803d-5cb94a356041-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:39.548383 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.547978 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:39.548383 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.547994 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:39.548383 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.548008 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3653569-1618-48e9-803d-5cb94a356041-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:39.548383 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.548022 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3653569-1618-48e9-803d-5cb94a356041-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:08:39.761647 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.761570 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a4949e1-4900-4ce8-9734-ce6e73bef410" path="/var/lib/kubelet/pods/9a4949e1-4900-4ce8-9734-ce6e73bef410/volumes" Oct 31 20:08:39.990252 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.990216 4910 generic.go:356] "Generic (PLEG): container finished" podID="c3653569-1618-48e9-803d-5cb94a356041" containerID="4e1129c1e6cf16ad431c16d9f3a49e3559a535c1eba14676af9fe0b87196a128" exitCode=0 Oct 31 20:08:39.990474 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.990266 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" Oct 31 20:08:39.990474 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.990308 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" event={"ID":"c3653569-1618-48e9-803d-5cb94a356041","Type":"ContainerDied","Data":"4e1129c1e6cf16ad431c16d9f3a49e3559a535c1eba14676af9fe0b87196a128"} Oct 31 20:08:39.990474 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.990374 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr" event={"ID":"c3653569-1618-48e9-803d-5cb94a356041","Type":"ContainerDied","Data":"53dc777aea019832d892256e970fc07c9b7b72ccdcbd10d8522603d6b8d202a3"} Oct 31 20:08:39.990474 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.990406 4910 scope.go:122] "RemoveContainer" containerID="4e1129c1e6cf16ad431c16d9f3a49e3559a535c1eba14676af9fe0b87196a128" Oct 31 20:08:40.000030 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:39.999992 4910 scope.go:122] "RemoveContainer" containerID="4e1129c1e6cf16ad431c16d9f3a49e3559a535c1eba14676af9fe0b87196a128" Oct 31 20:08:40.000307 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:08:40.000289 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e1129c1e6cf16ad431c16d9f3a49e3559a535c1eba14676af9fe0b87196a128\": container with ID starting with 4e1129c1e6cf16ad431c16d9f3a49e3559a535c1eba14676af9fe0b87196a128 not found: ID does not exist" containerID="4e1129c1e6cf16ad431c16d9f3a49e3559a535c1eba14676af9fe0b87196a128" Oct 31 20:08:40.000368 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:40.000315 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e1129c1e6cf16ad431c16d9f3a49e3559a535c1eba14676af9fe0b87196a128"} err="failed to get container status \"4e1129c1e6cf16ad431c16d9f3a49e3559a535c1eba14676af9fe0b87196a128\": rpc error: code = NotFound desc = could not find container \"4e1129c1e6cf16ad431c16d9f3a49e3559a535c1eba14676af9fe0b87196a128\": container with ID starting with 4e1129c1e6cf16ad431c16d9f3a49e3559a535c1eba14676af9fe0b87196a128 not found: ID does not exist" Oct 31 20:08:40.017892 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:40.017868 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr"] Oct 31 20:08:40.023088 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:40.023049 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-ph4dr"] Oct 31 20:08:41.763269 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:08:41.763212 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3653569-1618-48e9-803d-5cb94a356041" path="/var/lib/kubelet/pods/c3653569-1618-48e9-803d-5cb94a356041/volumes" Oct 31 20:08:42.713015 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:08:42.712977 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:09:09.320090 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320051 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf"] Oct 31 20:09:09.320606 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320463 4910 cpu_manager.go:414] "containerMap: removing container" podUID="55d3a124-18b2-4198-b3ee-7ae4180538f8" containerName="istio-proxy" Oct 31 20:09:09.320606 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320476 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55d3a124-18b2-4198-b3ee-7ae4180538f8" containerName="istio-proxy" Oct 31 20:09:09.320606 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320490 4910 cpu_manager.go:414] "containerMap: removing container" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" Oct 31 20:09:09.320606 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320494 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" Oct 31 20:09:09.320606 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320502 4910 cpu_manager.go:414] "containerMap: removing container" podUID="cb65a183-bd28-4a21-8895-82b7a39b4693" containerName="istio-proxy" Oct 31 20:09:09.320606 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320507 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb65a183-bd28-4a21-8895-82b7a39b4693" containerName="istio-proxy" Oct 31 20:09:09.320606 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320514 4910 cpu_manager.go:414] "containerMap: removing container" podUID="7d56bb36-ed0e-4127-9461-cbec40a46b95" containerName="payload-processing" Oct 31 20:09:09.320606 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320521 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d56bb36-ed0e-4127-9461-cbec40a46b95" containerName="payload-processing" Oct 31 20:09:09.320606 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320535 4910 cpu_manager.go:414] "containerMap: removing container" podUID="0e75dc23-b035-493b-9528-f24b18b140bd" containerName="revoke-keys" Oct 31 20:09:09.320606 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320540 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e75dc23-b035-493b-9528-f24b18b140bd" containerName="revoke-keys" Oct 31 20:09:09.320606 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320550 4910 cpu_manager.go:414] "containerMap: removing container" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" Oct 31 20:09:09.320606 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320555 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" Oct 31 20:09:09.320606 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320564 4910 cpu_manager.go:414] "containerMap: removing container" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" Oct 31 20:09:09.320606 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320570 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" Oct 31 20:09:09.320606 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320575 4910 cpu_manager.go:414] "containerMap: removing container" podUID="15e605d1-e3b9-4c0b-9b03-689c480dd89c" containerName="istio-proxy" Oct 31 20:09:09.320606 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320580 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15e605d1-e3b9-4c0b-9b03-689c480dd89c" containerName="istio-proxy" Oct 31 20:09:09.320606 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320589 4910 cpu_manager.go:414] "containerMap: removing container" podUID="9a4949e1-4900-4ce8-9734-ce6e73bef410" containerName="payload-pre-processing" Oct 31 20:09:09.320606 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320593 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a4949e1-4900-4ce8-9734-ce6e73bef410" containerName="payload-pre-processing" Oct 31 20:09:09.320606 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320602 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c34f06bd-3b50-4342-9719-bcccd4be955f" containerName="payload-processing" Oct 31 20:09:09.320606 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320606 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34f06bd-3b50-4342-9719-bcccd4be955f" containerName="payload-processing" Oct 31 20:09:09.320606 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320614 4910 cpu_manager.go:414] "containerMap: removing container" podUID="f1917335-8922-4b66-a5c4-ec463de8e3ca" containerName="istio-proxy" Oct 31 20:09:09.320606 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320619 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1917335-8922-4b66-a5c4-ec463de8e3ca" containerName="istio-proxy" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320627 4910 cpu_manager.go:414] "containerMap: removing container" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320631 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320642 4910 cpu_manager.go:414] "containerMap: removing container" podUID="20862e55-868f-40e8-af36-4f3e30cf9361" containerName="maas-api" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320647 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20862e55-868f-40e8-af36-4f3e30cf9361" containerName="maas-api" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320653 4910 cpu_manager.go:414] "containerMap: removing container" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320657 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320664 4910 cpu_manager.go:414] "containerMap: removing container" podUID="ce5603fa-39d7-4705-a0e3-85297ad53a78" containerName="istio-proxy" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320668 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce5603fa-39d7-4705-a0e3-85297ad53a78" containerName="istio-proxy" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320675 4910 cpu_manager.go:414] "containerMap: removing container" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320679 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320685 4910 cpu_manager.go:414] "containerMap: removing container" podUID="78d95c12-a91e-44a2-94a4-001d4c3ec257" containerName="istio-proxy" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320689 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78d95c12-a91e-44a2-94a4-001d4c3ec257" containerName="istio-proxy" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320696 4910 cpu_manager.go:414] "containerMap: removing container" podUID="0da37695-41cb-4698-9658-c5f7eb993220" containerName="istio-proxy" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320700 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da37695-41cb-4698-9658-c5f7eb993220" containerName="istio-proxy" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320707 4910 cpu_manager.go:414] "containerMap: removing container" podUID="c3653569-1618-48e9-803d-5cb94a356041" containerName="istio-proxy" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320711 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3653569-1618-48e9-803d-5cb94a356041" containerName="istio-proxy" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320718 4910 cpu_manager.go:414] "containerMap: removing container" podUID="2358e6a2-66b8-44b0-99b6-52b9c75e2cfe" containerName="maas-api" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320722 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2358e6a2-66b8-44b0-99b6-52b9c75e2cfe" containerName="maas-api" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320729 4910 cpu_manager.go:414] "containerMap: removing container" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320734 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320744 4910 cpu_manager.go:414] "containerMap: removing container" podUID="9136da5b-c568-412a-b262-eb85bb4cf6b6" containerName="istio-proxy" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320748 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9136da5b-c568-412a-b262-eb85bb4cf6b6" containerName="istio-proxy" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320806 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="204c09e1-cfce-4add-986d-165e78767839" containerName="istio-proxy" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320817 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="20862e55-868f-40e8-af36-4f3e30cf9361" containerName="maas-api" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320834 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="3803d62c-0b0f-49ce-a755-0db6506489d4" containerName="main" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320840 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce5603fa-39d7-4705-a0e3-85297ad53a78" containerName="istio-proxy" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320848 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="423e19ac-6ae0-4240-9a04-17068d61ebd7" containerName="main" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320856 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="78d95c12-a91e-44a2-94a4-001d4c3ec257" containerName="istio-proxy" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320865 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da37695-41cb-4698-9658-c5f7eb993220" containerName="istio-proxy" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320876 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3653569-1618-48e9-803d-5cb94a356041" containerName="istio-proxy" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320886 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb65a183-bd28-4a21-8895-82b7a39b4693" containerName="istio-proxy" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320895 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="2358e6a2-66b8-44b0-99b6-52b9c75e2cfe" containerName="maas-api" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320904 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="902b298d-646c-45e4-bff8-d505de51c689" containerName="main" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320926 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="9136da5b-c568-412a-b262-eb85bb4cf6b6" containerName="istio-proxy" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320935 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="55d3a124-18b2-4198-b3ee-7ae4180538f8" containerName="istio-proxy" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320943 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cfae09f-0cde-4452-9ac1-babca8bb72a6" containerName="main" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320950 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb65a183-bd28-4a21-8895-82b7a39b4693" containerName="istio-proxy" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320958 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d56bb36-ed0e-4127-9461-cbec40a46b95" containerName="payload-processing" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320964 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e75dc23-b035-493b-9528-f24b18b140bd" containerName="revoke-keys" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320970 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bfbef1c-402f-4a21-a378-e4906ec55c4c" containerName="main" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320978 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="d01fa9ef-252a-417c-b6d8-27375c4e1b90" containerName="main" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320983 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="15e605d1-e3b9-4c0b-9b03-689c480dd89c" containerName="istio-proxy" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320988 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a4949e1-4900-4ce8-9734-ce6e73bef410" containerName="payload-pre-processing" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320994 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34f06bd-3b50-4342-9719-bcccd4be955f" containerName="payload-processing" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.320999 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1917335-8922-4b66-a5c4-ec463de8e3ca" containerName="istio-proxy" Oct 31 20:09:09.321416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.321006 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="97560889-c009-4292-968a-58758bd2266b" containerName="main" Oct 31 20:09:09.324382 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.324359 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.334132 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.334107 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pflws\" (UniqueName: \"kubernetes.io/projected/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-kube-api-access-pflws\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.334252 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.334156 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.334252 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.334189 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.334252 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.334215 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.334252 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.334242 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.334462 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.334279 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.334462 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.334309 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.334462 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.334384 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.334569 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.334516 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.348783 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.348755 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm"] Oct 31 20:09:09.349208 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.349186 4910 cpu_manager.go:414] "containerMap: removing container" podUID="cb65a183-bd28-4a21-8895-82b7a39b4693" containerName="istio-proxy" Oct 31 20:09:09.349208 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.349208 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb65a183-bd28-4a21-8895-82b7a39b4693" containerName="istio-proxy" Oct 31 20:09:09.352505 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.352481 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf"] Oct 31 20:09:09.352631 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.352572 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.369513 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.369467 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm"] Oct 31 20:09:09.434956 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.434924 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.435148 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.434963 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.435148 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.435011 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.435148 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.435040 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqzcx\" (UniqueName: \"kubernetes.io/projected/2bebe97b-e6c1-40d7-a33b-a65f56d86494-kube-api-access-wqzcx\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.435148 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.435103 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.435395 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.435152 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.435395 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.435240 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.435395 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.435265 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.435395 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.435287 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.435395 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.435371 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.435754 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.435454 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pflws\" (UniqueName: \"kubernetes.io/projected/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-kube-api-access-pflws\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.435754 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.435512 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.435754 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.435546 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.435754 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.435514 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.435754 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.435587 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.435754 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.435617 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.435754 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.435641 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.436140 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.435786 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.436140 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.435842 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.436140 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.435952 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.436140 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.435984 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.436285 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.436187 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.436285 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.436193 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.437386 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.437366 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.437635 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.437616 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.448363 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.448331 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.448517 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.448502 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pflws\" (UniqueName: \"kubernetes.io/projected/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-kube-api-access-pflws\") pod \"maas-default-gateway-openshift-default-75d7c785f-qqmwf\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.536631 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.536589 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.536631 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.536632 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.536871 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.536658 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.536871 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.536684 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.536871 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.536737 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.536871 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.536761 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.536871 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.536790 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.536871 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.536863 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqzcx\" (UniqueName: \"kubernetes.io/projected/2bebe97b-e6c1-40d7-a33b-a65f56d86494-kube-api-access-wqzcx\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.537247 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.536887 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.537247 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.537124 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.537247 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.537152 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.537396 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.537253 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.537396 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.537283 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.537637 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.537619 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.539215 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.539187 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.539442 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.539424 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.549724 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.549700 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.549928 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.549849 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqzcx\" (UniqueName: \"kubernetes.io/projected/2bebe97b-e6c1-40d7-a33b-a65f56d86494-kube-api-access-wqzcx\") pod \"maas-default-gateway-openshift-default-75d7c785f-fhccm\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.642106 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.642017 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:09.670105 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.670072 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:09.926896 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.926862 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf"] Oct 31 20:09:09.927347 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:09:09.927318 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac8e6f6_e865_4d5e_a72f_e5d0a5f15ba2.slice/crio-e63ec4e2f66bc071890af707acd05ec032274b3b87c4351ffcc0c4a8fbd514ce WatchSource:0}: Error finding container e63ec4e2f66bc071890af707acd05ec032274b3b87c4351ffcc0c4a8fbd514ce: Status 404 returned error can't find the container with id e63ec4e2f66bc071890af707acd05ec032274b3b87c4351ffcc0c4a8fbd514ce Oct 31 20:09:09.929631 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.929597 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:09:09.929715 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.929666 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:09:09.929715 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.929696 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:09:09.965506 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:09:09.965482 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bebe97b_e6c1_40d7_a33b_a65f56d86494.slice/crio-c3ee15d61c585c44f0ac7ad1e95c3011cf12bda68a28b1a2b01225604be3ff04 WatchSource:0}: Error finding container c3ee15d61c585c44f0ac7ad1e95c3011cf12bda68a28b1a2b01225604be3ff04: Status 404 returned error can't find the container with id c3ee15d61c585c44f0ac7ad1e95c3011cf12bda68a28b1a2b01225604be3ff04 Oct 31 20:09:09.965506 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.965489 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm"] Oct 31 20:09:09.967754 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.967723 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:09:09.967866 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.967801 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:09:09.967866 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:09.967850 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:09:10.167656 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:10.167613 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" event={"ID":"2bebe97b-e6c1-40d7-a33b-a65f56d86494","Type":"ContainerStarted","Data":"dffbadbafcf976da41e126e996d2e3d80225dc402ebfffd2a0dbac75dd245bd8"} Oct 31 20:09:10.167866 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:10.167690 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" event={"ID":"2bebe97b-e6c1-40d7-a33b-a65f56d86494","Type":"ContainerStarted","Data":"c3ee15d61c585c44f0ac7ad1e95c3011cf12bda68a28b1a2b01225604be3ff04"} Oct 31 20:09:10.169889 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:10.169856 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" event={"ID":"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2","Type":"ContainerStarted","Data":"401f5500249647df2f7f7d9cf73fb240fd1a3d51ce9739a4afb88554812637db"} Oct 31 20:09:10.170046 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:10.169940 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" event={"ID":"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2","Type":"ContainerStarted","Data":"e63ec4e2f66bc071890af707acd05ec032274b3b87c4351ffcc0c4a8fbd514ce"} Oct 31 20:09:10.197827 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:10.196709 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" podStartSLOduration=1.196688109 podStartE2EDuration="1.196688109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:09:10.192533134 +0000 UTC m=+2984.555353508" watchObservedRunningTime="2026-10-31 20:09:10.196688109 +0000 UTC m=+2984.559508483" Oct 31 20:09:10.224891 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:10.224798 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" podStartSLOduration=1.224778961 podStartE2EDuration="1.224778961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:09:10.220862332 +0000 UTC m=+2984.583682720" watchObservedRunningTime="2026-10-31 20:09:10.224778961 +0000 UTC m=+2984.587599336" Oct 31 20:09:10.656935 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:10.647021 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:10.670414 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:10.670374 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:11.651080 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:11.651037 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-qqmwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 31 20:09:11.651412 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:11.651375 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" podUID="8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 31 20:09:11.672075 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:11.672040 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fhccm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body= Oct 31 20:09:11.672503 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:11.672131 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" podUID="2bebe97b-e6c1-40d7-a33b-a65f56d86494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" Oct 31 20:09:12.646936 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:12.646326 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-qqmwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 31 20:09:12.646936 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:12.646392 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" podUID="8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 31 20:09:12.670922 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:12.670865 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fhccm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body= Oct 31 20:09:12.671108 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:12.670982 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" podUID="2bebe97b-e6c1-40d7-a33b-a65f56d86494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" Oct 31 20:09:12.713270 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:09:12.713232 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:09:13.645439 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:13.645391 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-qqmwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 31 20:09:13.645725 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:13.645697 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" podUID="8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 31 20:09:13.670846 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:13.670808 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fhccm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body= Oct 31 20:09:13.671218 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:13.671160 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" podUID="2bebe97b-e6c1-40d7-a33b-a65f56d86494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" Oct 31 20:09:14.646935 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:14.646545 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-qqmwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 31 20:09:14.646935 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:14.646630 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" podUID="8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 31 20:09:14.670623 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:14.670579 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fhccm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body= Oct 31 20:09:14.670834 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:14.670633 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" podUID="2bebe97b-e6c1-40d7-a33b-a65f56d86494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" Oct 31 20:09:15.643937 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:15.643401 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-qqmwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 31 20:09:15.643937 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:15.643471 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" podUID="8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 31 20:09:15.670468 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:15.670423 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fhccm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body= Oct 31 20:09:15.670989 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:15.670473 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" podUID="2bebe97b-e6c1-40d7-a33b-a65f56d86494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" Oct 31 20:09:16.643579 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:16.643538 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-qqmwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Oct 31 20:09:16.643793 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:16.643648 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" podUID="8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Oct 31 20:09:16.671066 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:16.671028 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-fhccm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body= Oct 31 20:09:16.671454 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:16.671090 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" podUID="2bebe97b-e6c1-40d7-a33b-a65f56d86494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" Oct 31 20:09:17.029575 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:17.029468 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8ec13c2-7z45g"] Oct 31 20:09:17.031825 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:17.031791 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8ec13c2-7z45g"] Oct 31 20:09:17.756877 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:17.756824 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29295413-7152-4a06-8d91-1a7e6819df6c" path="/var/lib/kubelet/pods/29295413-7152-4a06-8d91-1a7e6819df6c/volumes" Oct 31 20:09:18.388502 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:18.388464 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:18.389330 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:18.389302 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:18.390530 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:18.390267 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:09:18.459275 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:18.459236 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:18.459894 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:18.459858 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:18.461268 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:18.461250 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:09:26.027707 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:26.027666 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b37081-57vcz"] Oct 31 20:09:26.031074 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:26.031038 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b37081-57vcz"] Oct 31 20:09:26.898369 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:26.898336 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 20:09:26.926797 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:26.926770 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 20:09:27.753278 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:27.753241 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4936de05-9195-4a0a-b44c-75049422759b" path="/var/lib/kubelet/pods/4936de05-9195-4a0a-b44c-75049422759b/volumes" Oct 31 20:09:32.267509 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:32.267470 4910 scope.go:122] "RemoveContainer" containerID="95b77623206c8d124c7987acebf617bfb113712d3eb32366c755107068ce9d79" Oct 31 20:09:34.435856 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:34.435803 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx"] Oct 31 20:09:34.436415 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:34.436036 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" podUID="65cd5c60-1a89-4cd0-a754-b04cd8e46c37" containerName="discovery" containerID="cri-o://3d3b3e752555bc488d151235c451fed4632b4cea2738177bad47cbe6f79982be" gracePeriod=30 Oct 31 20:09:34.710770 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:34.710745 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 20:09:34.794452 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:34.794407 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-token\" (UniqueName: \"kubernetes.io/projected/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-token\") pod \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " Oct 31 20:09:34.794627 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:34.794472 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-cacerts\" (UniqueName: \"kubernetes.io/secret/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-cacerts\") pod \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " Oct 31 20:09:34.794627 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:34.794516 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-local-certs\") pod \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " Oct 31 20:09:34.794627 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:34.794615 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-csr-dns-cert\") pod \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " Oct 31 20:09:34.794773 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:34.794660 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-csr-ca-configmap\") pod \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " Oct 31 20:09:34.794773 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:34.794719 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-kubeconfig\") pod \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " Oct 31 20:09:34.794895 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:34.794774 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-kube-api-access-fk9s5\" (UniqueName: \"kubernetes.io/projected/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-kube-api-access-fk9s5\") pod \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\" (UID: \"65cd5c60-1a89-4cd0-a754-b04cd8e46c37\") " Oct 31 20:09:34.795158 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:34.795125 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-csr-ca-configmap" pod "65cd5c60-1a89-4cd0-a754-b04cd8e46c37" (UID: "65cd5c60-1a89-4cd0-a754-b04cd8e46c37"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:09:34.797204 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:34.797162 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-kube-api-access-fk9s5" pod "65cd5c60-1a89-4cd0-a754-b04cd8e46c37" (UID: "65cd5c60-1a89-4cd0-a754-b04cd8e46c37"). InnerVolumeSpecName "kube-api-access-fk9s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:09:34.797319 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:34.797299 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-token" pod "65cd5c60-1a89-4cd0-a754-b04cd8e46c37" (UID: "65cd5c60-1a89-4cd0-a754-b04cd8e46c37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:09:34.797445 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:34.797318 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-csr-dns-cert" pod "65cd5c60-1a89-4cd0-a754-b04cd8e46c37" (UID: "65cd5c60-1a89-4cd0-a754-b04cd8e46c37"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:09:34.797445 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:34.797323 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-local-certs" pod "65cd5c60-1a89-4cd0-a754-b04cd8e46c37" (UID: "65cd5c60-1a89-4cd0-a754-b04cd8e46c37"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:09:34.797445 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:34.797402 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-cacerts" pod "65cd5c60-1a89-4cd0-a754-b04cd8e46c37" (UID: "65cd5c60-1a89-4cd0-a754-b04cd8e46c37"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:09:34.797445 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:34.797422 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-kubeconfig" pod "65cd5c60-1a89-4cd0-a754-b04cd8e46c37" (UID: "65cd5c60-1a89-4cd0-a754-b04cd8e46c37"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:09:34.896107 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:34.896074 4910 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-kubeconfig\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:09:34.896107 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:34.896103 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk9s5\" (UniqueName: \"kubernetes.io/projected/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-kube-api-access-fk9s5\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:09:34.896107 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:34.896115 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:09:34.896369 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:34.896123 4910 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-cacerts\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:09:34.896369 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:34.896132 4910 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-local-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:09:34.896369 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:34.896141 4910 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-csr-dns-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:09:34.896369 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:34.896151 4910 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/65cd5c60-1a89-4cd0-a754-b04cd8e46c37-istio-csr-ca-configmap\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:09:35.330079 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:35.330045 4910 generic.go:356] "Generic (PLEG): container finished" podID="65cd5c60-1a89-4cd0-a754-b04cd8e46c37" containerID="3d3b3e752555bc488d151235c451fed4632b4cea2738177bad47cbe6f79982be" exitCode=0 Oct 31 20:09:35.330266 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:35.330094 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" Oct 31 20:09:35.330266 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:35.330125 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" event={"ID":"65cd5c60-1a89-4cd0-a754-b04cd8e46c37","Type":"ContainerDied","Data":"3d3b3e752555bc488d151235c451fed4632b4cea2738177bad47cbe6f79982be"} Oct 31 20:09:35.330266 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:35.330175 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx" event={"ID":"65cd5c60-1a89-4cd0-a754-b04cd8e46c37","Type":"ContainerDied","Data":"e3f9fda7ec4597d403600f5aa7b65d69f1c75bb537f93dfb0b36c7662c8dd0a9"} Oct 31 20:09:35.330266 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:35.330192 4910 scope.go:122] "RemoveContainer" containerID="3d3b3e752555bc488d151235c451fed4632b4cea2738177bad47cbe6f79982be" Oct 31 20:09:35.343733 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:35.343714 4910 scope.go:122] "RemoveContainer" containerID="3d3b3e752555bc488d151235c451fed4632b4cea2738177bad47cbe6f79982be" Oct 31 20:09:35.344034 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:09:35.344014 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d3b3e752555bc488d151235c451fed4632b4cea2738177bad47cbe6f79982be\": container with ID starting with 3d3b3e752555bc488d151235c451fed4632b4cea2738177bad47cbe6f79982be not found: ID does not exist" containerID="3d3b3e752555bc488d151235c451fed4632b4cea2738177bad47cbe6f79982be" Oct 31 20:09:35.344105 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:35.344045 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d3b3e752555bc488d151235c451fed4632b4cea2738177bad47cbe6f79982be"} err="failed to get container status \"3d3b3e752555bc488d151235c451fed4632b4cea2738177bad47cbe6f79982be\": rpc error: code = NotFound desc = could not find container \"3d3b3e752555bc488d151235c451fed4632b4cea2738177bad47cbe6f79982be\": container with ID starting with 3d3b3e752555bc488d151235c451fed4632b4cea2738177bad47cbe6f79982be not found: ID does not exist" Oct 31 20:09:35.355482 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:35.355454 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx"] Oct 31 20:09:35.363123 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:35.363103 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhgqx"] Oct 31 20:09:35.753274 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:09:35.753194 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65cd5c60-1a89-4cd0-a754-b04cd8e46c37" path="/var/lib/kubelet/pods/65cd5c60-1a89-4cd0-a754-b04cd8e46c37/volumes" Oct 31 20:09:42.711958 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:09:42.711896 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:10:12.712176 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:10:12.712136 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:10:25.033153 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:10:25.033112 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1b2c5fc-n4b69"] Oct 31 20:10:25.035244 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:10:25.035219 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1b2c5fc-n4b69"] Oct 31 20:10:25.757665 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:10:25.757624 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="698675aa-5484-476d-9b7a-e677c312dcc3" path="/var/lib/kubelet/pods/698675aa-5484-476d-9b7a-e677c312dcc3/volumes" Oct 31 20:10:26.037692 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:10:26.037592 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d613866-7sb6w"] Oct 31 20:10:26.041226 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:10:26.041198 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d613866-7sb6w"] Oct 31 20:10:27.791604 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:10:27.791555 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad7d564e-a00c-4722-9849-dcbd5de16477" path="/var/lib/kubelet/pods/ad7d564e-a00c-4722-9849-dcbd5de16477/volumes" Oct 31 20:10:27.906211 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:10:27.906082 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-w7w5h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:10:27.906211 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:10:27.906160 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-w7w5h" podUID="f5b43015-708e-4ef2-883e-cc121fe1c8e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:10:32.307431 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:10:32.307398 4910 scope.go:122] "RemoveContainer" containerID="5326e0fc7a07048464248e1cd34399483ac8a2ee2c2aa641761b267262c21112" Oct 31 20:10:32.316706 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:10:32.316682 4910 scope.go:122] "RemoveContainer" containerID="835c4e4f25410d9b42208ee4f01c378e6d78d9b90e41f6990a925e370a908d81" Oct 31 20:10:32.325547 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:10:32.325522 4910 scope.go:122] "RemoveContainer" containerID="f724e4a457fafec6ba52165a08bd8006458e529ddc0e320c2291aec3a994eb08" Oct 31 20:10:32.336665 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:10:32.336642 4910 scope.go:122] "RemoveContainer" containerID="7748b672b8bcf3bc88c4d53b7a68dbce8ae7ea08910751ab8e2992e7089db3b5" Oct 31 20:10:42.711300 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:10:42.711259 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:10:56.057841 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:10:56.057799 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee17cfdc-nrwsf"] Oct 31 20:10:56.061680 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:10:56.061649 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee17cfdc-nrwsf"] Oct 31 20:10:57.752834 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:10:57.752797 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="901b6b03-183b-4819-a239-7ae5565e9eba" path="/var/lib/kubelet/pods/901b6b03-183b-4819-a239-7ae5565e9eba/volumes" Oct 31 20:11:05.040507 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:11:05.040470 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6668e56-vzzz8"] Oct 31 20:11:05.044903 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:11:05.044878 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6668e56-vzzz8"] Oct 31 20:11:05.753580 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:11:05.753545 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e75dc23-b035-493b-9528-f24b18b140bd" path="/var/lib/kubelet/pods/0e75dc23-b035-493b-9528-f24b18b140bd/volumes" Oct 31 20:11:12.712321 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:11:12.712284 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:11:32.375184 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:11:32.375147 4910 scope.go:122] "RemoveContainer" containerID="6c327f9d547fcc0ad8e3b455630f403134df2d1a3e7dbe56f115905cda2ed460" Oct 31 20:11:32.384850 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:11:32.384827 4910 scope.go:122] "RemoveContainer" containerID="2d9f858c83a797f3d13edd4bc8b583d9348df4516215e54b5b3e85956ac1a407" Oct 31 20:11:32.394354 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:11:32.394331 4910 scope.go:122] "RemoveContainer" containerID="a2260afb15cb0b3ef0e2a92d45210220cb807724d0dc4c57d9f55704d85fab50" Oct 31 20:11:32.402771 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:11:32.402750 4910 scope.go:122] "RemoveContainer" containerID="6f2f35fcbffe7fc149be7b4d6989bbb25b404ec32a6d10891bc94c2697bd625f" Oct 31 20:11:32.416146 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:11:32.416123 4910 scope.go:122] "RemoveContainer" containerID="91687e4d221e5ddfc0f0d3139b0752a7d173b4c49ba85ad39d8a321782491bea" Oct 31 20:11:42.718067 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:11:42.718017 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:12:12.783249 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:12:12.783186 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:12:32.468987 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:12:32.468949 4910 scope.go:122] "RemoveContainer" containerID="0d9dbffa13c8f19a72c6700025afc010569036dcd2de411335b5f2686a75619f" Oct 31 20:12:32.477691 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:12:32.477668 4910 scope.go:122] "RemoveContainer" containerID="a7f0291d1d6ade3d5cc0a813ff9502f9da70ffee0981dba5141ab0e92bd7ca07" Oct 31 20:12:32.485377 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:12:32.485356 4910 scope.go:122] "RemoveContainer" containerID="090386b2ee3d22eea123a6347fb90d6a13a4a1c5c7565db075c684b33a288715" Oct 31 20:12:32.493183 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:12:32.493161 4910 scope.go:122] "RemoveContainer" containerID="0bb99fa894b6ae8dfd8f40536e6730c045922777b68d607a2fe62fbb8977c6a7" Oct 31 20:12:42.712299 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:12:42.712257 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:13:12.711626 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:13:12.711585 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:13:42.714014 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:13:42.713965 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:14:12.713042 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:14:12.712961 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:14:26.932165 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:14:26.932134 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 20:14:26.968444 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:14:26.968416 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 20:14:42.712211 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:14:42.712173 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:15:12.712497 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:15:12.712452 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:15:42.711238 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:15:42.711157 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:16:12.713163 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:16:12.713124 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:16:42.712303 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:16:42.712260 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:17:12.712083 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:17:12.711997 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:17:42.712233 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:17:42.712188 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:18:12.712097 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:18:12.712054 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:18:42.712189 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:18:42.712153 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:18:55.078309 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:18:55.078271 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf"] Oct 31 20:18:55.078833 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:18:55.078525 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" podUID="8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2" containerName="istio-proxy" containerID="cri-o://401f5500249647df2f7f7d9cf73fb240fd1a3d51ce9739a4afb88554812637db" gracePeriod=30 Oct 31 20:19:00.323412 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.323384 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:19:00.434838 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.434746 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istiod-ca-cert\") pod \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " Oct 31 20:19:00.434838 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.434787 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-podinfo\") pod \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " Oct 31 20:19:00.434838 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.434812 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-envoy\") pod \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " Oct 31 20:19:00.435093 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.434967 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-workload-socket\") pod \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " Oct 31 20:19:00.435093 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.435018 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-workload-certs\") pod \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " Oct 31 20:19:00.435093 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.435052 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-token\" (UniqueName: \"kubernetes.io/projected/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-token\") pod \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " Oct 31 20:19:00.435093 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.435086 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-credential-socket\") pod \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " Oct 31 20:19:00.435223 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.435142 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-data\") pod \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " Oct 31 20:19:00.435223 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.435190 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istiod-ca-cert" pod "8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2" (UID: "8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:19:00.435223 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.435200 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-kube-api-access-pflws\" (UniqueName: \"kubernetes.io/projected/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-kube-api-access-pflws\") pod \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\" (UID: \"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2\") " Oct 31 20:19:00.435359 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.435341 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-workload-socket" pod "8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2" (UID: "8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:19:00.435418 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.435386 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-workload-certs" pod "8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2" (UID: "8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:19:00.435490 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.435465 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-credential-socket" pod "8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2" (UID: "8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:19:00.435634 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.435613 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:19:00.435719 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.435641 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:19:00.435719 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.435652 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-data" pod "8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2" (UID: "8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:19:00.435719 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.435662 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:19:00.435719 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.435680 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:19:00.437418 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.437289 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-podinfo" pod "8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2" (UID: "8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:19:00.437418 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.437371 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-kube-api-access-pflws" pod "8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2" (UID: "8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2"). InnerVolumeSpecName "kube-api-access-pflws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:19:00.437418 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.437389 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-token" pod "8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2" (UID: "8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:19:00.437655 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.437514 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-envoy" pod "8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2" (UID: "8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:19:00.536861 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.536823 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:19:00.536861 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.536858 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pflws\" (UniqueName: \"kubernetes.io/projected/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-kube-api-access-pflws\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:19:00.536861 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.536868 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:19:00.537192 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.536877 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:19:00.537192 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.536885 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:19:00.562285 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.562245 4910 generic.go:356] "Generic (PLEG): container finished" podID="8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2" containerID="401f5500249647df2f7f7d9cf73fb240fd1a3d51ce9739a4afb88554812637db" exitCode=0 Oct 31 20:19:00.562446 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.562309 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" Oct 31 20:19:00.562446 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.562336 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" event={"ID":"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2","Type":"ContainerDied","Data":"401f5500249647df2f7f7d9cf73fb240fd1a3d51ce9739a4afb88554812637db"} Oct 31 20:19:00.562446 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.562409 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf" event={"ID":"8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2","Type":"ContainerDied","Data":"e63ec4e2f66bc071890af707acd05ec032274b3b87c4351ffcc0c4a8fbd514ce"} Oct 31 20:19:00.562446 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.562434 4910 scope.go:122] "RemoveContainer" containerID="401f5500249647df2f7f7d9cf73fb240fd1a3d51ce9739a4afb88554812637db" Oct 31 20:19:00.572669 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.572646 4910 scope.go:122] "RemoveContainer" containerID="401f5500249647df2f7f7d9cf73fb240fd1a3d51ce9739a4afb88554812637db" Oct 31 20:19:00.572994 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:19:00.572972 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"401f5500249647df2f7f7d9cf73fb240fd1a3d51ce9739a4afb88554812637db\": container with ID starting with 401f5500249647df2f7f7d9cf73fb240fd1a3d51ce9739a4afb88554812637db not found: ID does not exist" containerID="401f5500249647df2f7f7d9cf73fb240fd1a3d51ce9739a4afb88554812637db" Oct 31 20:19:00.573073 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.573010 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"401f5500249647df2f7f7d9cf73fb240fd1a3d51ce9739a4afb88554812637db"} err="failed to get container status \"401f5500249647df2f7f7d9cf73fb240fd1a3d51ce9739a4afb88554812637db\": rpc error: code = NotFound desc = could not find container \"401f5500249647df2f7f7d9cf73fb240fd1a3d51ce9739a4afb88554812637db\": container with ID starting with 401f5500249647df2f7f7d9cf73fb240fd1a3d51ce9739a4afb88554812637db not found: ID does not exist" Oct 31 20:19:00.587077 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.587049 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf"] Oct 31 20:19:00.591635 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:00.591612 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qqmwf"] Oct 31 20:19:01.752859 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:01.752819 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2" path="/var/lib/kubelet/pods/8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2/volumes" Oct 31 20:19:10.130087 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:10.130049 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm"] Oct 31 20:19:10.130515 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:10.130259 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" podUID="2bebe97b-e6c1-40d7-a33b-a65f56d86494" containerName="istio-proxy" containerID="cri-o://dffbadbafcf976da41e126e996d2e3d80225dc402ebfffd2a0dbac75dd245bd8" gracePeriod=30 Oct 31 20:19:12.711795 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:19:12.711757 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:19:15.377905 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.377874 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:19:15.462461 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.462376 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istiod-ca-cert\") pod \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " Oct 31 20:19:15.462461 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.462424 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-envoy\") pod \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " Oct 31 20:19:15.462461 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.462452 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-workload-socket\") pod \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " Oct 31 20:19:15.462746 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.462502 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bebe97b-e6c1-40d7-a33b-a65f56d86494-kube-api-access-wqzcx\" (UniqueName: \"kubernetes.io/projected/2bebe97b-e6c1-40d7-a33b-a65f56d86494-kube-api-access-wqzcx\") pod \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " Oct 31 20:19:15.462746 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.462535 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-token\" (UniqueName: \"kubernetes.io/projected/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-token\") pod \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " Oct 31 20:19:15.462871 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.462771 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-workload-socket" pod "2bebe97b-e6c1-40d7-a33b-a65f56d86494" (UID: "2bebe97b-e6c1-40d7-a33b-a65f56d86494"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:19:15.462871 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.462817 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istiod-ca-cert" pod "2bebe97b-e6c1-40d7-a33b-a65f56d86494" (UID: "2bebe97b-e6c1-40d7-a33b-a65f56d86494"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:19:15.463011 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.462883 4910 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-workload-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:19:15.464780 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.464750 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bebe97b-e6c1-40d7-a33b-a65f56d86494-kube-api-access-wqzcx" pod "2bebe97b-e6c1-40d7-a33b-a65f56d86494" (UID: "2bebe97b-e6c1-40d7-a33b-a65f56d86494"). InnerVolumeSpecName "kube-api-access-wqzcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:19:15.464893 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.464785 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-token" pod "2bebe97b-e6c1-40d7-a33b-a65f56d86494" (UID: "2bebe97b-e6c1-40d7-a33b-a65f56d86494"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:19:15.464893 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.464790 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-envoy" pod "2bebe97b-e6c1-40d7-a33b-a65f56d86494" (UID: "2bebe97b-e6c1-40d7-a33b-a65f56d86494"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:19:15.564010 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.563971 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-workload-certs\") pod \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " Oct 31 20:19:15.564185 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.564023 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-data\") pod \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " Oct 31 20:19:15.564185 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.564056 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-podinfo\") pod \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " Oct 31 20:19:15.564185 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.564094 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-credential-socket\") pod \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\" (UID: \"2bebe97b-e6c1-40d7-a33b-a65f56d86494\") " Oct 31 20:19:15.564319 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.564290 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-workload-certs" pod "2bebe97b-e6c1-40d7-a33b-a65f56d86494" (UID: "2bebe97b-e6c1-40d7-a33b-a65f56d86494"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:19:15.564354 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.564303 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-data" pod "2bebe97b-e6c1-40d7-a33b-a65f56d86494" (UID: "2bebe97b-e6c1-40d7-a33b-a65f56d86494"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:19:15.564425 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.564404 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-credential-socket" pod "2bebe97b-e6c1-40d7-a33b-a65f56d86494" (UID: "2bebe97b-e6c1-40d7-a33b-a65f56d86494"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:19:15.564509 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.564496 4910 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-credential-socket\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:19:15.564549 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.564512 4910 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istiod-ca-cert\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:19:15.564549 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.564522 4910 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-envoy\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:19:15.564549 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.564531 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqzcx\" (UniqueName: \"kubernetes.io/projected/2bebe97b-e6c1-40d7-a33b-a65f56d86494-kube-api-access-wqzcx\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:19:15.564549 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.564541 4910 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-token\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:19:15.564549 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.564549 4910 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-workload-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:19:15.564714 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.564557 4910 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-data\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:19:15.566094 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.566075 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-podinfo" pod "2bebe97b-e6c1-40d7-a33b-a65f56d86494" (UID: "2bebe97b-e6c1-40d7-a33b-a65f56d86494"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 20:19:15.639126 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.639089 4910 generic.go:356] "Generic (PLEG): container finished" podID="2bebe97b-e6c1-40d7-a33b-a65f56d86494" containerID="dffbadbafcf976da41e126e996d2e3d80225dc402ebfffd2a0dbac75dd245bd8" exitCode=0 Oct 31 20:19:15.639305 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.639137 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" Oct 31 20:19:15.639305 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.639174 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" event={"ID":"2bebe97b-e6c1-40d7-a33b-a65f56d86494","Type":"ContainerDied","Data":"dffbadbafcf976da41e126e996d2e3d80225dc402ebfffd2a0dbac75dd245bd8"} Oct 31 20:19:15.639305 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.639267 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm" event={"ID":"2bebe97b-e6c1-40d7-a33b-a65f56d86494","Type":"ContainerDied","Data":"c3ee15d61c585c44f0ac7ad1e95c3011cf12bda68a28b1a2b01225604be3ff04"} Oct 31 20:19:15.639305 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.639297 4910 scope.go:122] "RemoveContainer" containerID="dffbadbafcf976da41e126e996d2e3d80225dc402ebfffd2a0dbac75dd245bd8" Oct 31 20:19:15.649093 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.649075 4910 scope.go:122] "RemoveContainer" containerID="dffbadbafcf976da41e126e996d2e3d80225dc402ebfffd2a0dbac75dd245bd8" Oct 31 20:19:15.649345 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:19:15.649327 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dffbadbafcf976da41e126e996d2e3d80225dc402ebfffd2a0dbac75dd245bd8\": container with ID starting with dffbadbafcf976da41e126e996d2e3d80225dc402ebfffd2a0dbac75dd245bd8 not found: ID does not exist" containerID="dffbadbafcf976da41e126e996d2e3d80225dc402ebfffd2a0dbac75dd245bd8" Oct 31 20:19:15.649414 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.649354 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dffbadbafcf976da41e126e996d2e3d80225dc402ebfffd2a0dbac75dd245bd8"} err="failed to get container status \"dffbadbafcf976da41e126e996d2e3d80225dc402ebfffd2a0dbac75dd245bd8\": rpc error: code = NotFound desc = could not find container \"dffbadbafcf976da41e126e996d2e3d80225dc402ebfffd2a0dbac75dd245bd8\": container with ID starting with dffbadbafcf976da41e126e996d2e3d80225dc402ebfffd2a0dbac75dd245bd8 not found: ID does not exist" Oct 31 20:19:15.665659 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.665433 4910 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bebe97b-e6c1-40d7-a33b-a65f56d86494-istio-podinfo\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:19:15.666460 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.666437 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm"] Oct 31 20:19:15.672482 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.672457 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-fhccm"] Oct 31 20:19:15.752518 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:15.752444 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bebe97b-e6c1-40d7-a33b-a65f56d86494" path="/var/lib/kubelet/pods/2bebe97b-e6c1-40d7-a33b-a65f56d86494/volumes" Oct 31 20:19:18.714369 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.714330 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl"] Oct 31 20:19:18.714808 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.714751 4910 cpu_manager.go:414] "containerMap: removing container" podUID="2bebe97b-e6c1-40d7-a33b-a65f56d86494" containerName="istio-proxy" Oct 31 20:19:18.714808 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.714765 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bebe97b-e6c1-40d7-a33b-a65f56d86494" containerName="istio-proxy" Oct 31 20:19:18.714808 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.714783 4910 cpu_manager.go:414] "containerMap: removing container" podUID="65cd5c60-1a89-4cd0-a754-b04cd8e46c37" containerName="discovery" Oct 31 20:19:18.714808 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.714788 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65cd5c60-1a89-4cd0-a754-b04cd8e46c37" containerName="discovery" Oct 31 20:19:18.714808 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.714809 4910 cpu_manager.go:414] "containerMap: removing container" podUID="8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2" containerName="istio-proxy" Oct 31 20:19:18.715021 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.714815 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2" containerName="istio-proxy" Oct 31 20:19:18.715021 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.714874 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac8e6f6-e865-4d5e-a72f-e5d0a5f15ba2" containerName="istio-proxy" Oct 31 20:19:18.715021 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.714884 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bebe97b-e6c1-40d7-a33b-a65f56d86494" containerName="istio-proxy" Oct 31 20:19:18.715021 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.714892 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="65cd5c60-1a89-4cd0-a754-b04cd8e46c37" containerName="discovery" Oct 31 20:19:18.719643 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.719617 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:19:18.725943 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.725875 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-844db7\"/\"default-dockercfg-xnn49\"" Oct 31 20:19:18.727137 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.727115 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-844db7\"/\"simulator-single-main-844db7-kserve-self-signed-certs\"" Oct 31 20:19:18.730804 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.730766 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-844db7\"/\"kube-root-ca.crt\"" Oct 31 20:19:18.731078 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.731049 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-844db7\"/\"openshift-service-ca.crt\"" Oct 31 20:19:18.732505 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.732481 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl"] Oct 31 20:19:18.789423 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.789387 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pst5\" (UniqueName: \"kubernetes.io/projected/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-kube-api-access-2pst5\") pod \"simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl\" (UID: \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\") " pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:19:18.789636 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.789440 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-model-cache\") pod \"simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl\" (UID: \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\") " pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:19:18.789636 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.789500 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-tmp-dir\") pod \"simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl\" (UID: \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\") " pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:19:18.789636 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.789608 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-home\") pod \"simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl\" (UID: \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\") " pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:19:18.789802 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.789648 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-dshm\") pod \"simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl\" (UID: \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\") " pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:19:18.789802 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.789673 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-tls-certs\") pod \"simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl\" (UID: \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\") " pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:19:18.891151 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.891112 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pst5\" (UniqueName: \"kubernetes.io/projected/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-kube-api-access-2pst5\") pod \"simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl\" (UID: \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\") " pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:19:18.891450 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.891178 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-model-cache\") pod \"simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl\" (UID: \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\") " pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:19:18.891450 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.891211 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-tmp-dir\") pod \"simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl\" (UID: \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\") " pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:19:18.891450 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.891290 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-home\") pod \"simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl\" (UID: \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\") " pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:19:18.891450 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.891323 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-dshm\") pod \"simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl\" (UID: \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\") " pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:19:18.891450 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.891339 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-tls-certs\") pod \"simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl\" (UID: \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\") " pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:19:18.891798 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.891728 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-model-cache\") pod \"simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl\" (UID: \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\") " pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:19:18.891798 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.891750 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-home\") pod \"simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl\" (UID: \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\") " pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:19:18.891798 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.891733 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-tmp-dir\") pod \"simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl\" (UID: \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\") " pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:19:18.893652 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.893625 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-dshm\") pod \"simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl\" (UID: \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\") " pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:19:18.893899 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.893874 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-tls-certs\") pod \"simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl\" (UID: \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\") " pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:19:18.906407 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:18.906385 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pst5\" (UniqueName: \"kubernetes.io/projected/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-kube-api-access-2pst5\") pod \"simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl\" (UID: \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\") " pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:19:19.035459 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:19.035366 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:19:19.323808 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:19.323776 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl"] Oct 31 20:19:19.324757 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:19:19.324724 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaccc8c3d_1a8b_4a1a_aaf7_82d2883475cb.slice/crio-9634742c0a140c7b2f28a97baae2e8e08adca0e5f04db44979fe0de1e64628cb WatchSource:0}: Error finding container 9634742c0a140c7b2f28a97baae2e8e08adca0e5f04db44979fe0de1e64628cb: Status 404 returned error can't find the container with id 9634742c0a140c7b2f28a97baae2e8e08adca0e5f04db44979fe0de1e64628cb Oct 31 20:19:19.663066 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:19.662983 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" event={"ID":"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb","Type":"ContainerStarted","Data":"a54bb051935b2f735937f30ac059ec7d5094e39d71ca7f9ae425a7cffee81ac5"} Oct 31 20:19:19.663066 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:19.663032 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" event={"ID":"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb","Type":"ContainerStarted","Data":"9634742c0a140c7b2f28a97baae2e8e08adca0e5f04db44979fe0de1e64628cb"} Oct 31 20:19:19.685480 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:19.685418 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" podStartSLOduration=1.685405474 podStartE2EDuration="1.685405474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:19:19.682572768 +0000 UTC m=+3594.045393139" watchObservedRunningTime="2026-10-31 20:19:19.685405474 +0000 UTC m=+3594.048225846" Oct 31 20:19:20.035747 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:20.035658 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:19:25.267190 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.267150 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs"] Oct 31 20:19:25.271076 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.271046 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.287158 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.287128 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs"] Oct 31 20:19:25.354175 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.354137 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.354175 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.354178 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.354427 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.354213 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.354427 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.354287 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.354427 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.354352 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.354427 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.354404 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thbqn\" (UniqueName: \"kubernetes.io/projected/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-kube-api-access-thbqn\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.354634 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.354441 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.354634 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.354471 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.354634 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.354499 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.455677 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.455636 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.455677 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.455677 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.455973 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.455801 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.455973 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.455841 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.455973 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.455899 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.455973 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.455965 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thbqn\" (UniqueName: \"kubernetes.io/projected/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-kube-api-access-thbqn\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.456216 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.455991 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.456216 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.456036 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.456300 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.456234 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.456422 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.456370 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.456549 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.456455 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.456549 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.456466 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.456634 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.456561 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.456697 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.456678 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.458073 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.458053 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.458367 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.458348 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.468933 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.468883 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.469817 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.469793 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thbqn\" (UniqueName: \"kubernetes.io/projected/574d9e49-82cb-4a4e-ac28-ce3174ea17ab-kube-api-access-thbqn\") pod \"maas-default-gateway-openshift-default-75d7c785f-qgfcs\" (UID: \"574d9e49-82cb-4a4e-ac28-ce3174ea17ab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.588081 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.588041 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:25.876533 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.876499 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs"] Oct 31 20:19:25.877465 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:19:25.877442 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod574d9e49_82cb_4a4e_ac28_ce3174ea17ab.slice/crio-d3a75529578864f7ea264fffd972ed563de23df6401122ccd3ba5d3c255505bf WatchSource:0}: Error finding container d3a75529578864f7ea264fffd972ed563de23df6401122ccd3ba5d3c255505bf: Status 404 returned error can't find the container with id d3a75529578864f7ea264fffd972ed563de23df6401122ccd3ba5d3c255505bf Oct 31 20:19:25.879703 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.879670 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:19:25.879798 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.879749 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:19:25.879859 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:25.879795 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:19:26.706220 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:26.706181 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" event={"ID":"574d9e49-82cb-4a4e-ac28-ce3174ea17ab","Type":"ContainerStarted","Data":"ba2e1d1f4e3bb6b4a10b5ad5cc41324f95c8c9db7ba68c4bd85cadb78862753d"} Oct 31 20:19:26.706686 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:26.706236 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" event={"ID":"574d9e49-82cb-4a4e-ac28-ce3174ea17ab","Type":"ContainerStarted","Data":"d3a75529578864f7ea264fffd972ed563de23df6401122ccd3ba5d3c255505bf"} Oct 31 20:19:26.747075 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:26.747004 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" podStartSLOduration=1.746988462 podStartE2EDuration="1.746988462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:19:26.743402787 +0000 UTC m=+3601.106223170" watchObservedRunningTime="2026-10-31 20:19:26.746988462 +0000 UTC m=+3601.109808835" Oct 31 20:19:26.968546 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:26.968463 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 20:19:27.006711 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:27.006245 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 20:19:27.588324 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:27.588289 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:27.590029 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:27.589996 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-qgfcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body= Oct 31 20:19:27.590185 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:27.590054 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" podUID="574d9e49-82cb-4a4e-ac28-ce3174ea17ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" Oct 31 20:19:28.588865 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:28.588822 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-qgfcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body= Oct 31 20:19:28.589325 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:28.588889 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" podUID="574d9e49-82cb-4a4e-ac28-ce3174ea17ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" Oct 31 20:19:29.038999 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:29.038296 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:19:29.054134 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:29.054101 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:19:29.588739 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:29.588702 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-qgfcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body= Oct 31 20:19:29.588963 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:29.588761 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" podUID="574d9e49-82cb-4a4e-ac28-ce3174ea17ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" Oct 31 20:19:29.756295 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:29.756260 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:19:30.588821 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:30.588783 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-qgfcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body= Oct 31 20:19:30.589041 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:30.588836 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" podUID="574d9e49-82cb-4a4e-ac28-ce3174ea17ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" Oct 31 20:19:31.588613 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:31.588561 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-qgfcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body= Oct 31 20:19:31.588846 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:31.588642 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" podUID="574d9e49-82cb-4a4e-ac28-ce3174ea17ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" Oct 31 20:19:32.589294 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:32.589267 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-qgfcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" start-of-body= Oct 31 20:19:32.589753 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:32.589316 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" podUID="574d9e49-82cb-4a4e-ac28-ce3174ea17ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": dial tcp 10.132.1.40:15021: connect: connection refused" Oct 31 20:19:34.085784 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:34.085749 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:34.086333 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:34.086035 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:34.087039 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:34.087014 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" Oct 31 20:19:40.241115 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.241080 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld"] Oct 31 20:19:40.246253 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.246217 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.257644 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.257615 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld"] Oct 31 20:19:40.296077 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.296037 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c7b75b-240c-4c56-be8a-3159c9a57f05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.296691 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.296663 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c7b75b-240c-4c56-be8a-3159c9a57f05-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.296823 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.296697 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g4t6\" (UniqueName: \"kubernetes.io/projected/24c7b75b-240c-4c56-be8a-3159c9a57f05-kube-api-access-4g4t6\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.296823 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.296767 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c7b75b-240c-4c56-be8a-3159c9a57f05-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.296823 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.296811 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c7b75b-240c-4c56-be8a-3159c9a57f05-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.296994 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.296839 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c7b75b-240c-4c56-be8a-3159c9a57f05-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.296994 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.296861 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c7b75b-240c-4c56-be8a-3159c9a57f05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.296994 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.296885 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c7b75b-240c-4c56-be8a-3159c9a57f05-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.296994 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.296936 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c7b75b-240c-4c56-be8a-3159c9a57f05-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.397650 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.397615 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c7b75b-240c-4c56-be8a-3159c9a57f05-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.397838 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.397661 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c7b75b-240c-4c56-be8a-3159c9a57f05-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.397838 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.397779 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c7b75b-240c-4c56-be8a-3159c9a57f05-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.397838 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.397832 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c7b75b-240c-4c56-be8a-3159c9a57f05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.398027 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.397868 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c7b75b-240c-4c56-be8a-3159c9a57f05-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.398027 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.397935 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c7b75b-240c-4c56-be8a-3159c9a57f05-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.398027 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.397971 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c7b75b-240c-4c56-be8a-3159c9a57f05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.398027 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.398010 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c7b75b-240c-4c56-be8a-3159c9a57f05-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.398327 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.398040 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g4t6\" (UniqueName: \"kubernetes.io/projected/24c7b75b-240c-4c56-be8a-3159c9a57f05-kube-api-access-4g4t6\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.398327 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.398088 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c7b75b-240c-4c56-be8a-3159c9a57f05-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.398448 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.398315 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24c7b75b-240c-4c56-be8a-3159c9a57f05-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.398448 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.398336 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24c7b75b-240c-4c56-be8a-3159c9a57f05-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.398448 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.398415 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24c7b75b-240c-4c56-be8a-3159c9a57f05-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.398638 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.398599 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24c7b75b-240c-4c56-be8a-3159c9a57f05-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.400143 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.400122 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24c7b75b-240c-4c56-be8a-3159c9a57f05-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.400625 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.400603 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24c7b75b-240c-4c56-be8a-3159c9a57f05-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.410159 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.410140 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24c7b75b-240c-4c56-be8a-3159c9a57f05-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.410241 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.410142 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g4t6\" (UniqueName: \"kubernetes.io/projected/24c7b75b-240c-4c56-be8a-3159c9a57f05-kube-api-access-4g4t6\") pod \"maas-default-gateway-openshift-default-75d7c785f-f92ld\" (UID: \"24c7b75b-240c-4c56-be8a-3159c9a57f05\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.561752 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.561720 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:40.840561 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.840532 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld"] Oct 31 20:19:40.842334 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:19:40.842300 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c7b75b_240c_4c56_be8a_3159c9a57f05.slice/crio-5356f78503f751289eba94e94b3b3e649b18257e03c2db29c4c820432b19f1c4 WatchSource:0}: Error finding container 5356f78503f751289eba94e94b3b3e649b18257e03c2db29c4c820432b19f1c4: Status 404 returned error can't find the container with id 5356f78503f751289eba94e94b3b3e649b18257e03c2db29c4c820432b19f1c4 Oct 31 20:19:40.844792 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.844759 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:19:40.844906 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.844839 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:19:40.844906 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:40.844883 4910 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 31 20:19:41.816954 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:41.816893 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" event={"ID":"24c7b75b-240c-4c56-be8a-3159c9a57f05","Type":"ContainerStarted","Data":"62bb493333868275f708771c74aba681c207299565fb6141bc923102068a1519"} Oct 31 20:19:41.817379 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:41.816979 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" event={"ID":"24c7b75b-240c-4c56-be8a-3159c9a57f05","Type":"ContainerStarted","Data":"5356f78503f751289eba94e94b3b3e649b18257e03c2db29c4c820432b19f1c4"} Oct 31 20:19:41.856949 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:41.856866 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" podStartSLOduration=1.8568452020000001 podStartE2EDuration="1.856845202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:19:41.85417679 +0000 UTC m=+3616.216997168" watchObservedRunningTime="2026-10-31 20:19:41.856845202 +0000 UTC m=+3616.219665574" Oct 31 20:19:42.562811 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:42.562765 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:42.564456 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:42.564431 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-f92ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Oct 31 20:19:42.564526 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:42.564496 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" podUID="24c7b75b-240c-4c56-be8a-3159c9a57f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Oct 31 20:19:42.712574 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:19:42.712535 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:19:43.562386 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:43.562350 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-f92ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Oct 31 20:19:43.562812 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:43.562417 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" podUID="24c7b75b-240c-4c56-be8a-3159c9a57f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Oct 31 20:19:44.562286 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:44.562250 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-f92ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Oct 31 20:19:44.562474 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:44.562305 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" podUID="24c7b75b-240c-4c56-be8a-3159c9a57f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Oct 31 20:19:45.562481 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:45.562442 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-f92ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Oct 31 20:19:45.562943 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:45.562553 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" podUID="24c7b75b-240c-4c56-be8a-3159c9a57f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Oct 31 20:19:46.562214 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:46.562176 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-f92ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Oct 31 20:19:46.562391 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:46.562233 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" podUID="24c7b75b-240c-4c56-be8a-3159c9a57f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Oct 31 20:19:47.562108 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:47.562070 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-f92ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Oct 31 20:19:47.562600 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:47.562130 4910 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" podUID="24c7b75b-240c-4c56-be8a-3159c9a57f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Oct 31 20:19:49.127435 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:49.127403 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:49.127933 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:49.127828 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:19:49.128795 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:19:49.128767 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" Oct 31 20:20:05.258556 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:05.258518 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl"] Oct 31 20:20:12.713156 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:20:12.713064 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:20:20.279219 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:20.279177 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" podUID="accc8c3d-1a8b-4a1a-aaf7-82d2883475cb" containerName="main" containerID="cri-o://a54bb051935b2f735937f30ac059ec7d5094e39d71ca7f9ae425a7cffee81ac5" gracePeriod=45 Oct 31 20:20:20.532697 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:20.532637 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:20:20.576936 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:20.576862 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-tls-certs\" (UniqueName: \"kubernetes.io/secret/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-tls-certs\") pod \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\" (UID: \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\") " Oct 31 20:20:20.577113 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:20.576944 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-home\" (UniqueName: \"kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-home\") pod \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\" (UID: \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\") " Oct 31 20:20:20.577113 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:20.576977 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-dshm\") pod \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\" (UID: \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\") " Oct 31 20:20:20.577113 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:20.577026 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-kube-api-access-2pst5\" (UniqueName: \"kubernetes.io/projected/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-kube-api-access-2pst5\") pod \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\" (UID: \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\") " Oct 31 20:20:20.577274 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:20.577117 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-model-cache\") pod \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\" (UID: \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\") " Oct 31 20:20:20.577274 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:20.577151 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-tmp-dir\") pod \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\" (UID: \"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb\") " Oct 31 20:20:20.578091 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:20.577860 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-tmp-dir" pod "accc8c3d-1a8b-4a1a-aaf7-82d2883475cb" (UID: "accc8c3d-1a8b-4a1a-aaf7-82d2883475cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:20:20.578314 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:20.578278 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-model-cache" pod "accc8c3d-1a8b-4a1a-aaf7-82d2883475cb" (UID: "accc8c3d-1a8b-4a1a-aaf7-82d2883475cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:20:20.578412 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:20.578333 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-home" pod "accc8c3d-1a8b-4a1a-aaf7-82d2883475cb" (UID: "accc8c3d-1a8b-4a1a-aaf7-82d2883475cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:20:20.582800 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:20.582760 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-kube-api-access-2pst5" pod "accc8c3d-1a8b-4a1a-aaf7-82d2883475cb" (UID: "accc8c3d-1a8b-4a1a-aaf7-82d2883475cb"). InnerVolumeSpecName "kube-api-access-2pst5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:20:20.583138 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:20.583115 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-tls-certs" pod "accc8c3d-1a8b-4a1a-aaf7-82d2883475cb" (UID: "accc8c3d-1a8b-4a1a-aaf7-82d2883475cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:20:20.595105 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:20.595064 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-dshm" pod "accc8c3d-1a8b-4a1a-aaf7-82d2883475cb" (UID: "accc8c3d-1a8b-4a1a-aaf7-82d2883475cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:20:20.678096 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:20.678055 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pst5\" (UniqueName: \"kubernetes.io/projected/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-kube-api-access-2pst5\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:20:20.678096 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:20.678091 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-model-cache\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:20:20.678313 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:20.678106 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-tmp-dir\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:20:20.678313 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:20.678120 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-tls-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:20:20.678313 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:20.678131 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-home\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:20:20.678313 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:20.678142 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb-dshm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:20:21.044145 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:21.044111 4910 generic.go:356] "Generic (PLEG): container finished" podID="accc8c3d-1a8b-4a1a-aaf7-82d2883475cb" containerID="a54bb051935b2f735937f30ac059ec7d5094e39d71ca7f9ae425a7cffee81ac5" exitCode=0 Oct 31 20:20:21.044361 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:21.044154 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" event={"ID":"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb","Type":"ContainerDied","Data":"a54bb051935b2f735937f30ac059ec7d5094e39d71ca7f9ae425a7cffee81ac5"} Oct 31 20:20:21.044361 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:21.044177 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" Oct 31 20:20:21.044361 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:21.044194 4910 scope.go:122] "RemoveContainer" containerID="a54bb051935b2f735937f30ac059ec7d5094e39d71ca7f9ae425a7cffee81ac5" Oct 31 20:20:21.044361 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:21.044185 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl" event={"ID":"accc8c3d-1a8b-4a1a-aaf7-82d2883475cb","Type":"ContainerDied","Data":"9634742c0a140c7b2f28a97baae2e8e08adca0e5f04db44979fe0de1e64628cb"} Oct 31 20:20:21.053560 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:21.053529 4910 scope.go:122] "RemoveContainer" containerID="a54bb051935b2f735937f30ac059ec7d5094e39d71ca7f9ae425a7cffee81ac5" Oct 31 20:20:21.053851 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:20:21.053822 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a54bb051935b2f735937f30ac059ec7d5094e39d71ca7f9ae425a7cffee81ac5\": container with ID starting with a54bb051935b2f735937f30ac059ec7d5094e39d71ca7f9ae425a7cffee81ac5 not found: ID does not exist" containerID="a54bb051935b2f735937f30ac059ec7d5094e39d71ca7f9ae425a7cffee81ac5" Oct 31 20:20:21.054008 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:21.053861 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54bb051935b2f735937f30ac059ec7d5094e39d71ca7f9ae425a7cffee81ac5"} err="failed to get container status \"a54bb051935b2f735937f30ac059ec7d5094e39d71ca7f9ae425a7cffee81ac5\": rpc error: code = NotFound desc = could not find container \"a54bb051935b2f735937f30ac059ec7d5094e39d71ca7f9ae425a7cffee81ac5\": container with ID starting with a54bb051935b2f735937f30ac059ec7d5094e39d71ca7f9ae425a7cffee81ac5 not found: ID does not exist" Oct 31 20:20:21.069131 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:21.069103 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl"] Oct 31 20:20:21.070956 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:21.070931 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-844db7/simulator-single-main-844db7-kserve-6495cd9fc7-hz9zl"] Oct 31 20:20:21.753183 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:21.753151 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="accc8c3d-1a8b-4a1a-aaf7-82d2883475cb" path="/var/lib/kubelet/pods/accc8c3d-1a8b-4a1a-aaf7-82d2883475cb/volumes" Oct 31 20:20:35.128153 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:35.128099 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-qgfcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:20:35.128779 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:35.128731 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" podUID="574d9e49-82cb-4a4e-ac28-ce3174ea17ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:20:35.172153 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:35.172089 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-f92ld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:20:35.172475 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:35.172439 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" podUID="24c7b75b-240c-4c56-be8a-3159c9a57f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:20:42.608532 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:42.608499 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m6g9g"] Oct 31 20:20:42.609059 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:42.608746 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m6g9g" podUID="54458929-81d7-4343-af60-fa19929a513e" containerName="manager" containerID="cri-o://ea3bcc90e7e40213921afb9dbc358d51cfd6967108122081a420f19429e47f61" gracePeriod=10 Oct 31 20:20:42.712004 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:20:42.711963 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:20:42.867826 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:42.867772 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m6g9g" Oct 31 20:20:43.003864 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:43.003826 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54458929-81d7-4343-af60-fa19929a513e-kube-api-access-26kdr\" (UniqueName: \"kubernetes.io/projected/54458929-81d7-4343-af60-fa19929a513e-kube-api-access-26kdr\") pod \"54458929-81d7-4343-af60-fa19929a513e\" (UID: \"54458929-81d7-4343-af60-fa19929a513e\") " Oct 31 20:20:43.004087 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:43.003901 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54458929-81d7-4343-af60-fa19929a513e-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/54458929-81d7-4343-af60-fa19929a513e-extensions-socket-volume\") pod \"54458929-81d7-4343-af60-fa19929a513e\" (UID: \"54458929-81d7-4343-af60-fa19929a513e\") " Oct 31 20:20:43.004329 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:43.004304 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54458929-81d7-4343-af60-fa19929a513e-extensions-socket-volume" pod "54458929-81d7-4343-af60-fa19929a513e" (UID: "54458929-81d7-4343-af60-fa19929a513e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:20:43.006039 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:43.006013 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54458929-81d7-4343-af60-fa19929a513e-kube-api-access-26kdr" pod "54458929-81d7-4343-af60-fa19929a513e" (UID: "54458929-81d7-4343-af60-fa19929a513e"). InnerVolumeSpecName "kube-api-access-26kdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:20:43.105026 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:43.104989 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26kdr\" (UniqueName: \"kubernetes.io/projected/54458929-81d7-4343-af60-fa19929a513e-kube-api-access-26kdr\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:20:43.105026 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:43.105020 4910 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/54458929-81d7-4343-af60-fa19929a513e-extensions-socket-volume\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:20:43.163438 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:43.163367 4910 generic.go:356] "Generic (PLEG): container finished" podID="54458929-81d7-4343-af60-fa19929a513e" containerID="ea3bcc90e7e40213921afb9dbc358d51cfd6967108122081a420f19429e47f61" exitCode=0 Oct 31 20:20:43.163438 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:43.163409 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m6g9g" Oct 31 20:20:43.163438 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:43.163433 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m6g9g" event={"ID":"54458929-81d7-4343-af60-fa19929a513e","Type":"ContainerDied","Data":"ea3bcc90e7e40213921afb9dbc358d51cfd6967108122081a420f19429e47f61"} Oct 31 20:20:43.163660 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:43.163473 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m6g9g" event={"ID":"54458929-81d7-4343-af60-fa19929a513e","Type":"ContainerDied","Data":"3d4023d5faf3adb1b53aed6a4c6fb9088362180d2af07f5f8f100d93ffea9c8c"} Oct 31 20:20:43.163660 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:43.163494 4910 scope.go:122] "RemoveContainer" containerID="ea3bcc90e7e40213921afb9dbc358d51cfd6967108122081a420f19429e47f61" Oct 31 20:20:43.175755 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:43.175734 4910 scope.go:122] "RemoveContainer" containerID="ea3bcc90e7e40213921afb9dbc358d51cfd6967108122081a420f19429e47f61" Oct 31 20:20:43.176071 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:20:43.176047 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea3bcc90e7e40213921afb9dbc358d51cfd6967108122081a420f19429e47f61\": container with ID starting with ea3bcc90e7e40213921afb9dbc358d51cfd6967108122081a420f19429e47f61 not found: ID does not exist" containerID="ea3bcc90e7e40213921afb9dbc358d51cfd6967108122081a420f19429e47f61" Oct 31 20:20:43.176125 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:43.176085 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea3bcc90e7e40213921afb9dbc358d51cfd6967108122081a420f19429e47f61"} err="failed to get container status \"ea3bcc90e7e40213921afb9dbc358d51cfd6967108122081a420f19429e47f61\": rpc error: code = NotFound desc = could not find container \"ea3bcc90e7e40213921afb9dbc358d51cfd6967108122081a420f19429e47f61\": container with ID starting with ea3bcc90e7e40213921afb9dbc358d51cfd6967108122081a420f19429e47f61 not found: ID does not exist" Oct 31 20:20:43.187520 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:43.187496 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m6g9g"] Oct 31 20:20:43.191736 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:43.191713 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-m6g9g"] Oct 31 20:20:43.753031 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:20:43.752998 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54458929-81d7-4343-af60-fa19929a513e" path="/var/lib/kubelet/pods/54458929-81d7-4343-af60-fa19929a513e/volumes" Oct 31 20:21:12.711426 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:21:12.711384 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:21:42.711667 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:21:42.711629 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:21:43.815265 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:21:43.815234 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g55sk"] Oct 31 20:21:43.815748 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:21:43.815627 4910 cpu_manager.go:414] "containerMap: removing container" podUID="54458929-81d7-4343-af60-fa19929a513e" containerName="manager" Oct 31 20:21:43.815748 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:21:43.815644 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54458929-81d7-4343-af60-fa19929a513e" containerName="manager" Oct 31 20:21:43.815748 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:21:43.815668 4910 cpu_manager.go:414] "containerMap: removing container" podUID="accc8c3d-1a8b-4a1a-aaf7-82d2883475cb" containerName="main" Oct 31 20:21:43.815748 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:21:43.815673 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="accc8c3d-1a8b-4a1a-aaf7-82d2883475cb" containerName="main" Oct 31 20:21:43.815889 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:21:43.815751 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="54458929-81d7-4343-af60-fa19929a513e" containerName="manager" Oct 31 20:21:43.815889 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:21:43.815764 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="accc8c3d-1a8b-4a1a-aaf7-82d2883475cb" containerName="main" Oct 31 20:21:43.819174 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:21:43.819158 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g55sk" Oct 31 20:21:43.833155 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:21:43.833131 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vj5f2\"" Oct 31 20:21:43.835321 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:21:43.835296 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g55sk"] Oct 31 20:21:43.942415 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:21:43.942376 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqtnm\" (UniqueName: \"kubernetes.io/projected/29e3a181-1107-4d40-bc55-e093391fc65f-kube-api-access-sqtnm\") pod \"kuadrant-operator-controller-manager-678d87766f-g55sk\" (UID: \"29e3a181-1107-4d40-bc55-e093391fc65f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g55sk" Oct 31 20:21:43.942629 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:21:43.942453 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29e3a181-1107-4d40-bc55-e093391fc65f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-g55sk\" (UID: \"29e3a181-1107-4d40-bc55-e093391fc65f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g55sk" Oct 31 20:21:44.043881 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:21:44.043845 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29e3a181-1107-4d40-bc55-e093391fc65f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-g55sk\" (UID: \"29e3a181-1107-4d40-bc55-e093391fc65f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g55sk" Oct 31 20:21:44.044081 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:21:44.043988 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqtnm\" (UniqueName: \"kubernetes.io/projected/29e3a181-1107-4d40-bc55-e093391fc65f-kube-api-access-sqtnm\") pod \"kuadrant-operator-controller-manager-678d87766f-g55sk\" (UID: \"29e3a181-1107-4d40-bc55-e093391fc65f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g55sk" Oct 31 20:21:44.044341 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:21:44.044315 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/29e3a181-1107-4d40-bc55-e093391fc65f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-g55sk\" (UID: \"29e3a181-1107-4d40-bc55-e093391fc65f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g55sk" Oct 31 20:21:44.056544 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:21:44.056519 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqtnm\" (UniqueName: \"kubernetes.io/projected/29e3a181-1107-4d40-bc55-e093391fc65f-kube-api-access-sqtnm\") pod \"kuadrant-operator-controller-manager-678d87766f-g55sk\" (UID: \"29e3a181-1107-4d40-bc55-e093391fc65f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g55sk" Oct 31 20:21:44.131336 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:21:44.131246 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g55sk" Oct 31 20:21:44.455479 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:21:44.455448 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g55sk"] Oct 31 20:21:44.455613 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:21:44.455568 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e3a181_1107_4d40_bc55_e093391fc65f.slice/crio-d2046fe6393c9ac7302ee21c6aed8449d8941c4935b958218f4a401cd24affe3 WatchSource:0}: Error finding container d2046fe6393c9ac7302ee21c6aed8449d8941c4935b958218f4a401cd24affe3: Status 404 returned error can't find the container with id d2046fe6393c9ac7302ee21c6aed8449d8941c4935b958218f4a401cd24affe3 Oct 31 20:21:44.488153 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:21:44.488114 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g55sk" event={"ID":"29e3a181-1107-4d40-bc55-e093391fc65f","Type":"ContainerStarted","Data":"d2046fe6393c9ac7302ee21c6aed8449d8941c4935b958218f4a401cd24affe3"} Oct 31 20:21:45.494852 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:21:45.494817 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g55sk" event={"ID":"29e3a181-1107-4d40-bc55-e093391fc65f","Type":"ContainerStarted","Data":"821dd2e6807ee2384d7e6bb549d176a16c1a72bb002910a45e7deb541aa4de9d"} Oct 31 20:21:45.495275 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:21:45.494964 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g55sk" Oct 31 20:21:45.532361 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:21:45.532314 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g55sk" podStartSLOduration=2.532301056 podStartE2EDuration="2.532301056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:21:45.530432527 +0000 UTC m=+3739.893252899" watchObservedRunningTime="2026-10-31 20:21:45.532301056 +0000 UTC m=+3739.895121423" Oct 31 20:21:56.502173 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:21:56.502143 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g55sk" Oct 31 20:22:05.141936 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:05.139196 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-qgfcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:22:05.141936 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:05.139283 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-qgfcs" podUID="574d9e49-82cb-4a4e-ac28-ce3174ea17ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:22:05.186936 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:05.186056 4910 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-f92ld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 20:22:05.186936 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:05.186158 4910 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-f92ld" podUID="24c7b75b-240c-4c56-be8a-3159c9a57f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 20:22:12.711716 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:22:12.711677 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:22:42.713574 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:22:42.713531 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:22:58.003495 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.003457 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg"] Oct 31 20:22:58.007133 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.007112 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:22:58.018416 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.018391 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-8407e9\"/\"simulator-single-main-8407e9-kserve-self-signed-certs\"" Oct 31 20:22:58.022574 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.022549 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-8407e9\"/\"default-dockercfg-4tv8t\"" Oct 31 20:22:58.023285 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.023265 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-8407e9\"/\"openshift-service-ca.crt\"" Oct 31 20:22:58.024009 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.023973 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-8407e9\"/\"kube-root-ca.crt\"" Oct 31 20:22:58.032844 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.032814 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg"] Oct 31 20:22:58.059169 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.059137 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-model-cache\") pod \"simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg\" (UID: \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\") " pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:22:58.059325 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.059173 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-home\") pod \"simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg\" (UID: \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\") " pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:22:58.059534 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.059514 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-tmp-dir\") pod \"simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg\" (UID: \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\") " pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:22:58.059631 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.059579 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkrzd\" (UniqueName: \"kubernetes.io/projected/4909d9f3-ecdc-4603-acc0-8911ea98c27f-kube-api-access-tkrzd\") pod \"simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg\" (UID: \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\") " pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:22:58.059631 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.059610 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4909d9f3-ecdc-4603-acc0-8911ea98c27f-tls-certs\") pod \"simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg\" (UID: \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\") " pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:22:58.059755 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.059655 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-dshm\") pod \"simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg\" (UID: \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\") " pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:22:58.160364 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.160326 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-model-cache\") pod \"simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg\" (UID: \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\") " pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:22:58.160364 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.160362 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-home\") pod \"simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg\" (UID: \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\") " pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:22:58.160630 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.160404 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-tmp-dir\") pod \"simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg\" (UID: \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\") " pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:22:58.160630 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.160456 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkrzd\" (UniqueName: \"kubernetes.io/projected/4909d9f3-ecdc-4603-acc0-8911ea98c27f-kube-api-access-tkrzd\") pod \"simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg\" (UID: \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\") " pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:22:58.160630 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.160584 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4909d9f3-ecdc-4603-acc0-8911ea98c27f-tls-certs\") pod \"simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg\" (UID: \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\") " pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:22:58.160879 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.160675 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-dshm\") pod \"simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg\" (UID: \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\") " pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:22:58.160879 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.160773 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-model-cache\") pod \"simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg\" (UID: \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\") " pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:22:58.160879 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.160828 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-tmp-dir\") pod \"simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg\" (UID: \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\") " pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:22:58.161090 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.160899 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-home\") pod \"simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg\" (UID: \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\") " pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:22:58.163100 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.163052 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-dshm\") pod \"simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg\" (UID: \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\") " pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:22:58.163294 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.163267 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4909d9f3-ecdc-4603-acc0-8911ea98c27f-tls-certs\") pod \"simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg\" (UID: \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\") " pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:22:58.173522 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.173495 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkrzd\" (UniqueName: \"kubernetes.io/projected/4909d9f3-ecdc-4603-acc0-8911ea98c27f-kube-api-access-tkrzd\") pod \"simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg\" (UID: \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\") " pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:22:58.324578 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.324537 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:22:58.602729 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.602695 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg"] Oct 31 20:22:58.603659 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:22:58.603630 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4909d9f3_ecdc_4603_acc0_8911ea98c27f.slice/crio-f97c02b06b0f7bf9840dfc30f28d6ab36d1c83dfeaf961dc0b67f980756d265e WatchSource:0}: Error finding container f97c02b06b0f7bf9840dfc30f28d6ab36d1c83dfeaf961dc0b67f980756d265e: Status 404 returned error can't find the container with id f97c02b06b0f7bf9840dfc30f28d6ab36d1c83dfeaf961dc0b67f980756d265e Oct 31 20:22:58.896501 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.896405 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" event={"ID":"4909d9f3-ecdc-4603-acc0-8911ea98c27f","Type":"ContainerStarted","Data":"523cf3bfdd7d4e962b208d0d2c421e587e908f3d6c51af092d17cb2334a17943"} Oct 31 20:22:58.896501 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.896455 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" event={"ID":"4909d9f3-ecdc-4603-acc0-8911ea98c27f","Type":"ContainerStarted","Data":"f97c02b06b0f7bf9840dfc30f28d6ab36d1c83dfeaf961dc0b67f980756d265e"} Oct 31 20:22:58.917693 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:58.917640 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" podStartSLOduration=1.9176253060000001 podStartE2EDuration="1.917625306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:22:58.916084591 +0000 UTC m=+3813.278904963" watchObservedRunningTime="2026-10-31 20:22:58.917625306 +0000 UTC m=+3813.280445679" Oct 31 20:22:59.325244 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:22:59.325191 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:23:08.324864 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:08.324780 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:23:08.337421 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:08.337393 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:23:08.963058 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:08.963018 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:23:12.711946 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:23:12.711894 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:23:28.780412 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:28.780377 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7801e99-sdxbg"] Oct 31 20:23:28.783681 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:28.783660 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7801e99-sdxbg" Oct 31 20:23:28.791541 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:28.791491 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:23:28.793435 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:28.792089 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 20:23:28.796177 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:28.796156 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7801e99-sdxbg"] Oct 31 20:23:28.798678 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:28.798655 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 20:23:28.798799 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:28.798735 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 20:23:28.949265 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:28.949214 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48f18f27-a847-459a-ac41-64524cbf5df4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7801e99-sdxbg\" (UID: \"48f18f27-a847-459a-ac41-64524cbf5df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7801e99-sdxbg" Oct 31 20:23:28.949466 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:28.949281 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48f18f27-a847-459a-ac41-64524cbf5df4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7801e99-sdxbg\" (UID: \"48f18f27-a847-459a-ac41-64524cbf5df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7801e99-sdxbg" Oct 31 20:23:28.949466 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:28.949437 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb856\" (UniqueName: \"kubernetes.io/projected/48f18f27-a847-459a-ac41-64524cbf5df4-kube-api-access-nb856\") pod \"maas-api-revoke-sub-b7801e99-sdxbg\" (UID: \"48f18f27-a847-459a-ac41-64524cbf5df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7801e99-sdxbg" Oct 31 20:23:29.050045 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:29.049952 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48f18f27-a847-459a-ac41-64524cbf5df4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7801e99-sdxbg\" (UID: \"48f18f27-a847-459a-ac41-64524cbf5df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7801e99-sdxbg" Oct 31 20:23:29.050045 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:29.050009 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48f18f27-a847-459a-ac41-64524cbf5df4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7801e99-sdxbg\" (UID: \"48f18f27-a847-459a-ac41-64524cbf5df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7801e99-sdxbg" Oct 31 20:23:29.050250 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:29.050078 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb856\" (UniqueName: \"kubernetes.io/projected/48f18f27-a847-459a-ac41-64524cbf5df4-kube-api-access-nb856\") pod \"maas-api-revoke-sub-b7801e99-sdxbg\" (UID: \"48f18f27-a847-459a-ac41-64524cbf5df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7801e99-sdxbg" Oct 31 20:23:29.050635 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:29.050611 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48f18f27-a847-459a-ac41-64524cbf5df4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7801e99-sdxbg\" (UID: \"48f18f27-a847-459a-ac41-64524cbf5df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7801e99-sdxbg" Oct 31 20:23:29.052540 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:29.052514 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48f18f27-a847-459a-ac41-64524cbf5df4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7801e99-sdxbg\" (UID: \"48f18f27-a847-459a-ac41-64524cbf5df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7801e99-sdxbg" Oct 31 20:23:29.063213 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:29.063180 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb856\" (UniqueName: \"kubernetes.io/projected/48f18f27-a847-459a-ac41-64524cbf5df4-kube-api-access-nb856\") pod \"maas-api-revoke-sub-b7801e99-sdxbg\" (UID: \"48f18f27-a847-459a-ac41-64524cbf5df4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7801e99-sdxbg" Oct 31 20:23:29.099252 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:29.099223 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7801e99-sdxbg" Oct 31 20:23:29.375632 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:29.375601 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7801e99-sdxbg"] Oct 31 20:23:29.376669 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:23:29.376637 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f18f27_a847_459a_ac41_64524cbf5df4.slice/crio-8388120fbc9850b60b8e66424cff0538c13ecf2dba482221bacc993ba9ecd612 WatchSource:0}: Error finding container 8388120fbc9850b60b8e66424cff0538c13ecf2dba482221bacc993ba9ecd612: Status 404 returned error can't find the container with id 8388120fbc9850b60b8e66424cff0538c13ecf2dba482221bacc993ba9ecd612 Oct 31 20:23:30.073445 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:30.073408 4910 generic.go:356] "Generic (PLEG): container finished" podID="48f18f27-a847-459a-ac41-64524cbf5df4" containerID="bf33f7e2beff4d84e16fa3a1103c6ac49f69561629391bd69b5314ea8c83ca60" exitCode=0 Oct 31 20:23:30.073865 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:30.073487 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7801e99-sdxbg" event={"ID":"48f18f27-a847-459a-ac41-64524cbf5df4","Type":"ContainerDied","Data":"bf33f7e2beff4d84e16fa3a1103c6ac49f69561629391bd69b5314ea8c83ca60"} Oct 31 20:23:30.073865 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:30.073542 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7801e99-sdxbg" event={"ID":"48f18f27-a847-459a-ac41-64524cbf5df4","Type":"ContainerStarted","Data":"8388120fbc9850b60b8e66424cff0538c13ecf2dba482221bacc993ba9ecd612"} Oct 31 20:23:31.219413 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:31.219386 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7801e99-sdxbg" Oct 31 20:23:31.373881 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:31.373798 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48f18f27-a847-459a-ac41-64524cbf5df4-kube-api-access-nb856\" (UniqueName: \"kubernetes.io/projected/48f18f27-a847-459a-ac41-64524cbf5df4-kube-api-access-nb856\") pod \"48f18f27-a847-459a-ac41-64524cbf5df4\" (UID: \"48f18f27-a847-459a-ac41-64524cbf5df4\") " Oct 31 20:23:31.373881 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:31.373872 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48f18f27-a847-459a-ac41-64524cbf5df4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48f18f27-a847-459a-ac41-64524cbf5df4-maas-api-xks-ca\") pod \"48f18f27-a847-459a-ac41-64524cbf5df4\" (UID: \"48f18f27-a847-459a-ac41-64524cbf5df4\") " Oct 31 20:23:31.374108 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:31.374011 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48f18f27-a847-459a-ac41-64524cbf5df4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48f18f27-a847-459a-ac41-64524cbf5df4-maas-api-openshift-ca\") pod \"48f18f27-a847-459a-ac41-64524cbf5df4\" (UID: \"48f18f27-a847-459a-ac41-64524cbf5df4\") " Oct 31 20:23:31.374408 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:31.374378 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48f18f27-a847-459a-ac41-64524cbf5df4-maas-api-openshift-ca" pod "48f18f27-a847-459a-ac41-64524cbf5df4" (UID: "48f18f27-a847-459a-ac41-64524cbf5df4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:23:31.376043 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:31.376017 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48f18f27-a847-459a-ac41-64524cbf5df4-maas-api-xks-ca" pod "48f18f27-a847-459a-ac41-64524cbf5df4" (UID: "48f18f27-a847-459a-ac41-64524cbf5df4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:23:31.376176 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:31.376066 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f18f27-a847-459a-ac41-64524cbf5df4-kube-api-access-nb856" pod "48f18f27-a847-459a-ac41-64524cbf5df4" (UID: "48f18f27-a847-459a-ac41-64524cbf5df4"). InnerVolumeSpecName "kube-api-access-nb856". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:23:31.475049 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:31.475014 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48f18f27-a847-459a-ac41-64524cbf5df4-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:31.475049 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:31.475044 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48f18f27-a847-459a-ac41-64524cbf5df4-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:31.475049 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:31.475055 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb856\" (UniqueName: \"kubernetes.io/projected/48f18f27-a847-459a-ac41-64524cbf5df4-kube-api-access-nb856\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:32.084446 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:32.084412 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7801e99-sdxbg" event={"ID":"48f18f27-a847-459a-ac41-64524cbf5df4","Type":"ContainerDied","Data":"8388120fbc9850b60b8e66424cff0538c13ecf2dba482221bacc993ba9ecd612"} Oct 31 20:23:32.084661 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:32.084454 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7801e99-sdxbg" Oct 31 20:23:32.084661 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:32.084465 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8388120fbc9850b60b8e66424cff0538c13ecf2dba482221bacc993ba9ecd612" Oct 31 20:23:39.743052 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:39.743010 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8"] Oct 31 20:23:39.743603 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:39.743585 4910 cpu_manager.go:414] "containerMap: removing container" podUID="48f18f27-a847-459a-ac41-64524cbf5df4" containerName="revoke-keys" Oct 31 20:23:39.743649 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:39.743605 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48f18f27-a847-459a-ac41-64524cbf5df4" containerName="revoke-keys" Oct 31 20:23:39.743742 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:39.743730 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="48f18f27-a847-459a-ac41-64524cbf5df4" containerName="revoke-keys" Oct 31 20:23:39.747544 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:39.747521 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8" Oct 31 20:23:39.754817 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:39.754426 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 20:23:39.754817 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:39.754483 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 20:23:39.754817 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:39.754486 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 20:23:39.755118 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:39.755000 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\"" Oct 31 20:23:39.770759 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:39.770731 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8"] Oct 31 20:23:39.853132 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:39.853090 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm7kd\" (UniqueName: \"kubernetes.io/projected/39ececdd-6b72-4663-b3ca-a01b270a73dc-kube-api-access-pm7kd\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8\" (UID: \"39ececdd-6b72-4663-b3ca-a01b270a73dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8" Oct 31 20:23:39.853324 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:39.853228 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ececdd-6b72-4663-b3ca-a01b270a73dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8\" (UID: \"39ececdd-6b72-4663-b3ca-a01b270a73dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8" Oct 31 20:23:39.853324 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:39.853307 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ececdd-6b72-4663-b3ca-a01b270a73dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8\" (UID: \"39ececdd-6b72-4663-b3ca-a01b270a73dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8" Oct 31 20:23:39.954935 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:39.954875 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm7kd\" (UniqueName: \"kubernetes.io/projected/39ececdd-6b72-4663-b3ca-a01b270a73dc-kube-api-access-pm7kd\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8\" (UID: \"39ececdd-6b72-4663-b3ca-a01b270a73dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8" Oct 31 20:23:39.955326 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:39.955296 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ececdd-6b72-4663-b3ca-a01b270a73dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8\" (UID: \"39ececdd-6b72-4663-b3ca-a01b270a73dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8" Oct 31 20:23:39.955469 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:39.955381 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ececdd-6b72-4663-b3ca-a01b270a73dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8\" (UID: \"39ececdd-6b72-4663-b3ca-a01b270a73dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8" Oct 31 20:23:39.956055 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:39.956025 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ececdd-6b72-4663-b3ca-a01b270a73dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8\" (UID: \"39ececdd-6b72-4663-b3ca-a01b270a73dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8" Oct 31 20:23:39.957805 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:39.957783 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ececdd-6b72-4663-b3ca-a01b270a73dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8\" (UID: \"39ececdd-6b72-4663-b3ca-a01b270a73dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8" Oct 31 20:23:39.967758 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:39.967733 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm7kd\" (UniqueName: \"kubernetes.io/projected/39ececdd-6b72-4663-b3ca-a01b270a73dc-kube-api-access-pm7kd\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8\" (UID: \"39ececdd-6b72-4663-b3ca-a01b270a73dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8" Oct 31 20:23:40.071306 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:40.071268 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8" Oct 31 20:23:40.368989 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:40.368961 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8"] Oct 31 20:23:40.370522 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:23:40.370494 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ececdd_6b72_4663_b3ca_a01b270a73dc.slice/crio-6f7e505690ab8d35b04fdcb65b8896ce058aada09bb10fbd6602d8676029d120 WatchSource:0}: Error finding container 6f7e505690ab8d35b04fdcb65b8896ce058aada09bb10fbd6602d8676029d120: Status 404 returned error can't find the container with id 6f7e505690ab8d35b04fdcb65b8896ce058aada09bb10fbd6602d8676029d120 Oct 31 20:23:41.156562 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:41.156517 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8" event={"ID":"39ececdd-6b72-4663-b3ca-a01b270a73dc","Type":"ContainerStarted","Data":"79f0283d38699e5e65ce98cc5e25be47dcdac237d91fbaed3c608929df9e7cc1"} Oct 31 20:23:41.156987 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:41.156587 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8" event={"ID":"39ececdd-6b72-4663-b3ca-a01b270a73dc","Type":"ContainerStarted","Data":"6f7e505690ab8d35b04fdcb65b8896ce058aada09bb10fbd6602d8676029d120"} Oct 31 20:23:41.178925 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:41.178871 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8" podStartSLOduration=2.178858212 podStartE2EDuration="2.178858212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:23:41.175704978 +0000 UTC m=+3855.538525350" watchObservedRunningTime="2026-10-31 20:23:41.178858212 +0000 UTC m=+3855.541678584" Oct 31 20:23:42.163728 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:42.163684 4910 generic.go:356] "Generic (PLEG): container finished" podID="39ececdd-6b72-4663-b3ca-a01b270a73dc" containerID="79f0283d38699e5e65ce98cc5e25be47dcdac237d91fbaed3c608929df9e7cc1" exitCode=0 Oct 31 20:23:42.164233 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:42.163788 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8" event={"ID":"39ececdd-6b72-4663-b3ca-a01b270a73dc","Type":"ContainerDied","Data":"79f0283d38699e5e65ce98cc5e25be47dcdac237d91fbaed3c608929df9e7cc1"} Oct 31 20:23:42.712793 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:23:42.712758 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:23:43.306154 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:43.306129 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8" Oct 31 20:23:43.391228 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:43.391191 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ececdd-6b72-4663-b3ca-a01b270a73dc-kube-api-access-pm7kd\" (UniqueName: \"kubernetes.io/projected/39ececdd-6b72-4663-b3ca-a01b270a73dc-kube-api-access-pm7kd\") pod \"39ececdd-6b72-4663-b3ca-a01b270a73dc\" (UID: \"39ececdd-6b72-4663-b3ca-a01b270a73dc\") " Oct 31 20:23:43.391444 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:43.391250 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39ececdd-6b72-4663-b3ca-a01b270a73dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ececdd-6b72-4663-b3ca-a01b270a73dc-maas-api-openshift-ca\") pod \"39ececdd-6b72-4663-b3ca-a01b270a73dc\" (UID: \"39ececdd-6b72-4663-b3ca-a01b270a73dc\") " Oct 31 20:23:43.391444 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:43.391347 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39ececdd-6b72-4663-b3ca-a01b270a73dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ececdd-6b72-4663-b3ca-a01b270a73dc-maas-api-xks-ca\") pod \"39ececdd-6b72-4663-b3ca-a01b270a73dc\" (UID: \"39ececdd-6b72-4663-b3ca-a01b270a73dc\") " Oct 31 20:23:43.391661 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:43.391632 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ececdd-6b72-4663-b3ca-a01b270a73dc-maas-api-openshift-ca" pod "39ececdd-6b72-4663-b3ca-a01b270a73dc" (UID: "39ececdd-6b72-4663-b3ca-a01b270a73dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 20:23:43.393783 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:43.393752 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39ececdd-6b72-4663-b3ca-a01b270a73dc-maas-api-xks-ca" pod "39ececdd-6b72-4663-b3ca-a01b270a73dc" (UID: "39ececdd-6b72-4663-b3ca-a01b270a73dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:23:43.394105 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:43.394075 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ececdd-6b72-4663-b3ca-a01b270a73dc-kube-api-access-pm7kd" pod "39ececdd-6b72-4663-b3ca-a01b270a73dc" (UID: "39ececdd-6b72-4663-b3ca-a01b270a73dc"). InnerVolumeSpecName "kube-api-access-pm7kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:23:43.492713 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:43.492625 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm7kd\" (UniqueName: \"kubernetes.io/projected/39ececdd-6b72-4663-b3ca-a01b270a73dc-kube-api-access-pm7kd\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:43.492713 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:43.492668 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ececdd-6b72-4663-b3ca-a01b270a73dc-maas-api-openshift-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:43.492713 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:43.492679 4910 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ececdd-6b72-4663-b3ca-a01b270a73dc-maas-api-xks-ca\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:44.176411 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:44.176380 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8" Oct 31 20:23:44.176603 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:44.176414 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8" event={"ID":"39ececdd-6b72-4663-b3ca-a01b270a73dc","Type":"ContainerDied","Data":"6f7e505690ab8d35b04fdcb65b8896ce058aada09bb10fbd6602d8676029d120"} Oct 31 20:23:44.176603 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:44.176463 4910 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f7e505690ab8d35b04fdcb65b8896ce058aada09bb10fbd6602d8676029d120" Oct 31 20:23:44.506655 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:44.506570 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg"] Oct 31 20:23:50.390092 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:50.390048 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8"] Oct 31 20:23:50.393872 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:50.393851 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8407e9-6bfq8"] Oct 31 20:23:51.753507 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:51.753460 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39ececdd-6b72-4663-b3ca-a01b270a73dc" path="/var/lib/kubelet/pods/39ececdd-6b72-4663-b3ca-a01b270a73dc/volumes" Oct 31 20:23:59.526744 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:59.526701 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" podUID="4909d9f3-ecdc-4603-acc0-8911ea98c27f" containerName="main" containerID="cri-o://523cf3bfdd7d4e962b208d0d2c421e587e908f3d6c51af092d17cb2334a17943" gracePeriod=45 Oct 31 20:23:59.779603 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:59.779546 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:23:59.842801 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:59.842750 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-model-cache\") pod \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\" (UID: \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\") " Oct 31 20:23:59.843098 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:59.842814 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-home\" (UniqueName: \"kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-home\") pod \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\" (UID: \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\") " Oct 31 20:23:59.843098 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:59.842882 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4909d9f3-ecdc-4603-acc0-8911ea98c27f-kube-api-access-tkrzd\" (UniqueName: \"kubernetes.io/projected/4909d9f3-ecdc-4603-acc0-8911ea98c27f-kube-api-access-tkrzd\") pod \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\" (UID: \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\") " Oct 31 20:23:59.843098 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:59.842936 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-tmp-dir\") pod \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\" (UID: \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\") " Oct 31 20:23:59.843098 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:59.843000 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4909d9f3-ecdc-4603-acc0-8911ea98c27f-tls-certs\" (UniqueName: \"kubernetes.io/secret/4909d9f3-ecdc-4603-acc0-8911ea98c27f-tls-certs\") pod \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\" (UID: \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\") " Oct 31 20:23:59.843098 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:59.843068 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-dshm\") pod \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\" (UID: \"4909d9f3-ecdc-4603-acc0-8911ea98c27f\") " Oct 31 20:23:59.843399 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:59.843091 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-model-cache" pod "4909d9f3-ecdc-4603-acc0-8911ea98c27f" (UID: "4909d9f3-ecdc-4603-acc0-8911ea98c27f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:23:59.843399 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:59.843106 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-home" pod "4909d9f3-ecdc-4603-acc0-8911ea98c27f" (UID: "4909d9f3-ecdc-4603-acc0-8911ea98c27f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:23:59.843399 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:59.843328 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-tmp-dir" pod "4909d9f3-ecdc-4603-acc0-8911ea98c27f" (UID: "4909d9f3-ecdc-4603-acc0-8911ea98c27f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:23:59.843577 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:59.843425 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-tmp-dir\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:59.843577 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:59.843444 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-model-cache\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:59.843577 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:59.843457 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-home\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:59.845287 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:59.845260 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4909d9f3-ecdc-4603-acc0-8911ea98c27f-kube-api-access-tkrzd" pod "4909d9f3-ecdc-4603-acc0-8911ea98c27f" (UID: "4909d9f3-ecdc-4603-acc0-8911ea98c27f"). InnerVolumeSpecName "kube-api-access-tkrzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:23:59.845694 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:59.845665 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4909d9f3-ecdc-4603-acc0-8911ea98c27f-tls-certs" pod "4909d9f3-ecdc-4603-acc0-8911ea98c27f" (UID: "4909d9f3-ecdc-4603-acc0-8911ea98c27f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:23:59.845849 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:59.845737 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-dshm" pod "4909d9f3-ecdc-4603-acc0-8911ea98c27f" (UID: "4909d9f3-ecdc-4603-acc0-8911ea98c27f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:23:59.944561 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:59.944525 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkrzd\" (UniqueName: \"kubernetes.io/projected/4909d9f3-ecdc-4603-acc0-8911ea98c27f-kube-api-access-tkrzd\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:59.944561 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:59.944556 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4909d9f3-ecdc-4603-acc0-8911ea98c27f-tls-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:23:59.944561 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:23:59.944566 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4909d9f3-ecdc-4603-acc0-8911ea98c27f-dshm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:24:00.271540 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:00.271504 4910 generic.go:356] "Generic (PLEG): container finished" podID="4909d9f3-ecdc-4603-acc0-8911ea98c27f" containerID="523cf3bfdd7d4e962b208d0d2c421e587e908f3d6c51af092d17cb2334a17943" exitCode=0 Oct 31 20:24:00.271743 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:00.271558 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" Oct 31 20:24:00.271743 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:00.271581 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" event={"ID":"4909d9f3-ecdc-4603-acc0-8911ea98c27f","Type":"ContainerDied","Data":"523cf3bfdd7d4e962b208d0d2c421e587e908f3d6c51af092d17cb2334a17943"} Oct 31 20:24:00.271743 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:00.271634 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg" event={"ID":"4909d9f3-ecdc-4603-acc0-8911ea98c27f","Type":"ContainerDied","Data":"f97c02b06b0f7bf9840dfc30f28d6ab36d1c83dfeaf961dc0b67f980756d265e"} Oct 31 20:24:00.271743 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:00.271663 4910 scope.go:122] "RemoveContainer" containerID="523cf3bfdd7d4e962b208d0d2c421e587e908f3d6c51af092d17cb2334a17943" Oct 31 20:24:00.281898 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:00.281874 4910 scope.go:122] "RemoveContainer" containerID="523cf3bfdd7d4e962b208d0d2c421e587e908f3d6c51af092d17cb2334a17943" Oct 31 20:24:00.282195 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:24:00.282172 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"523cf3bfdd7d4e962b208d0d2c421e587e908f3d6c51af092d17cb2334a17943\": container with ID starting with 523cf3bfdd7d4e962b208d0d2c421e587e908f3d6c51af092d17cb2334a17943 not found: ID does not exist" containerID="523cf3bfdd7d4e962b208d0d2c421e587e908f3d6c51af092d17cb2334a17943" Oct 31 20:24:00.282270 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:00.282203 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"523cf3bfdd7d4e962b208d0d2c421e587e908f3d6c51af092d17cb2334a17943"} err="failed to get container status \"523cf3bfdd7d4e962b208d0d2c421e587e908f3d6c51af092d17cb2334a17943\": rpc error: code = NotFound desc = could not find container \"523cf3bfdd7d4e962b208d0d2c421e587e908f3d6c51af092d17cb2334a17943\": container with ID starting with 523cf3bfdd7d4e962b208d0d2c421e587e908f3d6c51af092d17cb2334a17943 not found: ID does not exist" Oct 31 20:24:00.295493 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:00.295469 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg"] Oct 31 20:24:00.297360 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:00.297340 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-8407e9/simulator-single-main-8407e9-kserve-bcfbdcfdf-p4rzg"] Oct 31 20:24:01.753318 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:01.753284 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4909d9f3-ecdc-4603-acc0-8911ea98c27f" path="/var/lib/kubelet/pods/4909d9f3-ecdc-4603-acc0-8911ea98c27f/volumes" Oct 31 20:24:03.129540 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.129503 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh"] Oct 31 20:24:03.130226 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.130199 4910 cpu_manager.go:414] "containerMap: removing container" podUID="39ececdd-6b72-4663-b3ca-a01b270a73dc" containerName="revoke-keys" Oct 31 20:24:03.130226 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.130225 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ececdd-6b72-4663-b3ca-a01b270a73dc" containerName="revoke-keys" Oct 31 20:24:03.130387 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.130256 4910 cpu_manager.go:414] "containerMap: removing container" podUID="4909d9f3-ecdc-4603-acc0-8911ea98c27f" containerName="main" Oct 31 20:24:03.130387 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.130264 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4909d9f3-ecdc-4603-acc0-8911ea98c27f" containerName="main" Oct 31 20:24:03.130387 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.130368 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="39ececdd-6b72-4663-b3ca-a01b270a73dc" containerName="revoke-keys" Oct 31 20:24:03.130387 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.130385 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="4909d9f3-ecdc-4603-acc0-8911ea98c27f" containerName="main" Oct 31 20:24:03.134033 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.134010 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:24:03.161083 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.161062 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 20:24:03.161227 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.161108 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 20:24:03.163210 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.163195 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5hpmb\"" Oct 31 20:24:03.163287 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.163277 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-098205f1-kserve-self-signed-certs\"" Oct 31 20:24:03.178995 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.178971 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh"] Oct 31 20:24:03.276849 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.276811 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-dshm\") pod \"e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh\" (UID: \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\") " pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:24:03.276849 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.276845 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-tmp-dir\") pod \"e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh\" (UID: \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\") " pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:24:03.277100 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.276871 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-model-cache\") pod \"e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh\" (UID: \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\") " pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:24:03.277100 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.276949 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q995k\" (UniqueName: \"kubernetes.io/projected/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-kube-api-access-q995k\") pod \"e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh\" (UID: \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\") " pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:24:03.277100 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.277061 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-home\") pod \"e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh\" (UID: \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\") " pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:24:03.277100 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.277090 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-tls-certs\") pod \"e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh\" (UID: \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\") " pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:24:03.378519 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.378486 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-dshm\") pod \"e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh\" (UID: \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\") " pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:24:03.378519 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.378521 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-tmp-dir\") pod \"e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh\" (UID: \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\") " pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:24:03.378738 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.378545 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-model-cache\") pod \"e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh\" (UID: \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\") " pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:24:03.378738 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.378657 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q995k\" (UniqueName: \"kubernetes.io/projected/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-kube-api-access-q995k\") pod \"e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh\" (UID: \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\") " pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:24:03.378738 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.378717 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-home\") pod \"e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh\" (UID: \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\") " pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:24:03.378870 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.378743 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-tls-certs\") pod \"e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh\" (UID: \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\") " pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:24:03.378966 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.378947 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-tmp-dir\") pod \"e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh\" (UID: \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\") " pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:24:03.379024 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.379004 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-model-cache\") pod \"e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh\" (UID: \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\") " pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:24:03.379092 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.379070 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-home\") pod \"e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh\" (UID: \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\") " pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:24:03.380890 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.380829 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-dshm\") pod \"e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh\" (UID: \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\") " pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:24:03.381113 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.381097 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-tls-certs\") pod \"e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh\" (UID: \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\") " pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:24:03.391479 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.391453 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q995k\" (UniqueName: \"kubernetes.io/projected/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-kube-api-access-q995k\") pod \"e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh\" (UID: \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\") " pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:24:03.446499 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.446464 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:24:03.726737 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:03.726707 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh"] Oct 31 20:24:03.728476 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:24:03.728446 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod607f14a8_50f6_4f95_9a86_e2c93b5ed2a0.slice/crio-b65359436fdb231c156e2f9ec9f796a94923e3a7b8f2f6b6b5091089200b0c2a WatchSource:0}: Error finding container b65359436fdb231c156e2f9ec9f796a94923e3a7b8f2f6b6b5091089200b0c2a: Status 404 returned error can't find the container with id b65359436fdb231c156e2f9ec9f796a94923e3a7b8f2f6b6b5091089200b0c2a Oct 31 20:24:04.295435 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:04.295382 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" event={"ID":"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0","Type":"ContainerStarted","Data":"8c5c419609c1808da41734880b951cbf0d1ed889e6a77980f8e916fbfadaebef"} Oct 31 20:24:04.295823 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:04.295443 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" event={"ID":"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0","Type":"ContainerStarted","Data":"b65359436fdb231c156e2f9ec9f796a94923e3a7b8f2f6b6b5091089200b0c2a"} Oct 31 20:24:04.315621 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:04.315569 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" podStartSLOduration=1.315551636 podStartE2EDuration="1.315551636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:24:04.315265485 +0000 UTC m=+3878.678085858" watchObservedRunningTime="2026-10-31 20:24:04.315551636 +0000 UTC m=+3878.678372010" Oct 31 20:24:04.447506 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:04.447455 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:24:12.711973 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:24:12.711927 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:24:13.447070 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:13.447033 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:24:13.459644 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:13.459616 4910 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:24:14.389403 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:14.389368 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:24:27.003443 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:27.003410 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 20:24:27.065480 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:27.065455 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 20:24:42.711366 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:24:42.711288 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:24:46.732757 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:24:46.732723 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh"] Oct 31 20:25:01.758777 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:01.758738 4910 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" podUID="607f14a8-50f6-4f95-9a86-e2c93b5ed2a0" containerName="main" containerID="cri-o://8c5c419609c1808da41734880b951cbf0d1ed889e6a77980f8e916fbfadaebef" gracePeriod=45 Oct 31 20:25:02.012759 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.012687 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:25:02.173321 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.173285 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-home\" (UniqueName: \"kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-home\") pod \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\" (UID: \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\") " Oct 31 20:25:02.173499 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.173330 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-tmp-dir\") pod \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\" (UID: \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\") " Oct 31 20:25:02.173499 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.173370 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-kube-api-access-q995k\" (UniqueName: \"kubernetes.io/projected/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-kube-api-access-q995k\") pod \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\" (UID: \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\") " Oct 31 20:25:02.173499 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.173400 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-tls-certs\" (UniqueName: \"kubernetes.io/secret/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-tls-certs\") pod \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\" (UID: \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\") " Oct 31 20:25:02.173499 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.173462 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-dshm\") pod \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\" (UID: \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\") " Oct 31 20:25:02.173764 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.173499 4910 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-model-cache\") pod \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\" (UID: \"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0\") " Oct 31 20:25:02.173764 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.173620 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-tmp-dir" pod "607f14a8-50f6-4f95-9a86-e2c93b5ed2a0" (UID: "607f14a8-50f6-4f95-9a86-e2c93b5ed2a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:25:02.173764 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.173654 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-home" pod "607f14a8-50f6-4f95-9a86-e2c93b5ed2a0" (UID: "607f14a8-50f6-4f95-9a86-e2c93b5ed2a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:25:02.173970 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.173855 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-model-cache" pod "607f14a8-50f6-4f95-9a86-e2c93b5ed2a0" (UID: "607f14a8-50f6-4f95-9a86-e2c93b5ed2a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:25:02.173970 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.173885 4910 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-home\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:25:02.173970 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.173904 4910 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-tmp-dir\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:25:02.175761 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.175735 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-dshm" pod "607f14a8-50f6-4f95-9a86-e2c93b5ed2a0" (UID: "607f14a8-50f6-4f95-9a86-e2c93b5ed2a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 20:25:02.175867 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.175826 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-kube-api-access-q995k" pod "607f14a8-50f6-4f95-9a86-e2c93b5ed2a0" (UID: "607f14a8-50f6-4f95-9a86-e2c93b5ed2a0"). InnerVolumeSpecName "kube-api-access-q995k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 20:25:02.175867 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.175841 4910 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-tls-certs" pod "607f14a8-50f6-4f95-9a86-e2c93b5ed2a0" (UID: "607f14a8-50f6-4f95-9a86-e2c93b5ed2a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 20:25:02.275245 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.275160 4910 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q995k\" (UniqueName: \"kubernetes.io/projected/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-kube-api-access-q995k\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:25:02.275245 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.275199 4910 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-tls-certs\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:25:02.275245 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.275215 4910 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-dshm\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:25:02.275245 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.275229 4910 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0-model-cache\") on node \"ip-10-0-142-116.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 20:25:02.702380 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.702343 4910 generic.go:356] "Generic (PLEG): container finished" podID="607f14a8-50f6-4f95-9a86-e2c93b5ed2a0" containerID="8c5c419609c1808da41734880b951cbf0d1ed889e6a77980f8e916fbfadaebef" exitCode=0 Oct 31 20:25:02.702594 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.702400 4910 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" Oct 31 20:25:02.702594 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.702430 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" event={"ID":"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0","Type":"ContainerDied","Data":"8c5c419609c1808da41734880b951cbf0d1ed889e6a77980f8e916fbfadaebef"} Oct 31 20:25:02.702594 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.702489 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh" event={"ID":"607f14a8-50f6-4f95-9a86-e2c93b5ed2a0","Type":"ContainerDied","Data":"b65359436fdb231c156e2f9ec9f796a94923e3a7b8f2f6b6b5091089200b0c2a"} Oct 31 20:25:02.702594 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.702512 4910 scope.go:122] "RemoveContainer" containerID="8c5c419609c1808da41734880b951cbf0d1ed889e6a77980f8e916fbfadaebef" Oct 31 20:25:02.712495 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.712478 4910 scope.go:122] "RemoveContainer" containerID="8c5c419609c1808da41734880b951cbf0d1ed889e6a77980f8e916fbfadaebef" Oct 31 20:25:02.712767 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:25:02.712747 4910 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c5c419609c1808da41734880b951cbf0d1ed889e6a77980f8e916fbfadaebef\": container with ID starting with 8c5c419609c1808da41734880b951cbf0d1ed889e6a77980f8e916fbfadaebef not found: ID does not exist" containerID="8c5c419609c1808da41734880b951cbf0d1ed889e6a77980f8e916fbfadaebef" Oct 31 20:25:02.712822 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.712777 4910 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c5c419609c1808da41734880b951cbf0d1ed889e6a77980f8e916fbfadaebef"} err="failed to get container status \"8c5c419609c1808da41734880b951cbf0d1ed889e6a77980f8e916fbfadaebef\": rpc error: code = NotFound desc = could not find container \"8c5c419609c1808da41734880b951cbf0d1ed889e6a77980f8e916fbfadaebef\": container with ID starting with 8c5c419609c1808da41734880b951cbf0d1ed889e6a77980f8e916fbfadaebef not found: ID does not exist" Oct 31 20:25:02.725842 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.725811 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh"] Oct 31 20:25:02.731951 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:02.731921 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-098205f1-kserve-58cd4f675c-bfsfh"] Oct 31 20:25:03.775545 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:03.775508 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="607f14a8-50f6-4f95-9a86-e2c93b5ed2a0" path="/var/lib/kubelet/pods/607f14a8-50f6-4f95-9a86-e2c93b5ed2a0/volumes" Oct 31 20:25:12.712714 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:25:12.712657 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:25:32.776018 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:32.775979 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7fd9fcc7c9-dmvph"] Oct 31 20:25:32.776427 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:32.776382 4910 cpu_manager.go:414] "containerMap: removing container" podUID="607f14a8-50f6-4f95-9a86-e2c93b5ed2a0" containerName="main" Oct 31 20:25:32.776427 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:32.776394 4910 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="607f14a8-50f6-4f95-9a86-e2c93b5ed2a0" containerName="main" Oct 31 20:25:32.776502 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:32.776475 4910 memory_manager.go:365] "RemoveStaleState removing state" podUID="607f14a8-50f6-4f95-9a86-e2c93b5ed2a0" containerName="main" Oct 31 20:25:32.780695 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:32.780677 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-dmvph" Oct 31 20:25:32.793961 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:32.793934 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7fd9fcc7c9-dmvph"] Oct 31 20:25:32.805742 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:32.805705 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvzbq\" (UniqueName: \"kubernetes.io/projected/4dd374d8-2dc8-45ff-9f9f-a82c019259af-kube-api-access-pvzbq\") pod \"maas-api-7fd9fcc7c9-dmvph\" (UID: \"4dd374d8-2dc8-45ff-9f9f-a82c019259af\") " pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-dmvph" Oct 31 20:25:32.805874 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:32.805749 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd374d8-2dc8-45ff-9f9f-a82c019259af-metrics-certs\") pod \"maas-api-7fd9fcc7c9-dmvph\" (UID: \"4dd374d8-2dc8-45ff-9f9f-a82c019259af\") " pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-dmvph" Oct 31 20:25:32.805874 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:32.805778 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dd374d8-2dc8-45ff-9f9f-a82c019259af-maas-api-tls\") pod \"maas-api-7fd9fcc7c9-dmvph\" (UID: \"4dd374d8-2dc8-45ff-9f9f-a82c019259af\") " pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-dmvph" Oct 31 20:25:32.806022 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:32.805899 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dd374d8-2dc8-45ff-9f9f-a82c019259af-openshift-service-ca\") pod \"maas-api-7fd9fcc7c9-dmvph\" (UID: \"4dd374d8-2dc8-45ff-9f9f-a82c019259af\") " pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-dmvph" Oct 31 20:25:32.809877 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:32.809853 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 20:25:32.809877 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:32.809870 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 20:25:32.810043 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:32.809876 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9dsr5\"" Oct 31 20:25:32.817043 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:32.817011 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 20:25:32.817160 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:32.817081 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 20:25:32.906756 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:32.906682 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dd374d8-2dc8-45ff-9f9f-a82c019259af-openshift-service-ca\") pod \"maas-api-7fd9fcc7c9-dmvph\" (UID: \"4dd374d8-2dc8-45ff-9f9f-a82c019259af\") " pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-dmvph" Oct 31 20:25:32.906756 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:32.906729 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvzbq\" (UniqueName: \"kubernetes.io/projected/4dd374d8-2dc8-45ff-9f9f-a82c019259af-kube-api-access-pvzbq\") pod \"maas-api-7fd9fcc7c9-dmvph\" (UID: \"4dd374d8-2dc8-45ff-9f9f-a82c019259af\") " pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-dmvph" Oct 31 20:25:32.906756 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:32.906749 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd374d8-2dc8-45ff-9f9f-a82c019259af-metrics-certs\") pod \"maas-api-7fd9fcc7c9-dmvph\" (UID: \"4dd374d8-2dc8-45ff-9f9f-a82c019259af\") " pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-dmvph" Oct 31 20:25:32.907079 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:32.907051 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dd374d8-2dc8-45ff-9f9f-a82c019259af-maas-api-tls\") pod \"maas-api-7fd9fcc7c9-dmvph\" (UID: \"4dd374d8-2dc8-45ff-9f9f-a82c019259af\") " pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-dmvph" Oct 31 20:25:32.907623 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:32.907598 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dd374d8-2dc8-45ff-9f9f-a82c019259af-openshift-service-ca\") pod \"maas-api-7fd9fcc7c9-dmvph\" (UID: \"4dd374d8-2dc8-45ff-9f9f-a82c019259af\") " pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-dmvph" Oct 31 20:25:32.909361 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:32.909335 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4dd374d8-2dc8-45ff-9f9f-a82c019259af-maas-api-tls\") pod \"maas-api-7fd9fcc7c9-dmvph\" (UID: \"4dd374d8-2dc8-45ff-9f9f-a82c019259af\") " pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-dmvph" Oct 31 20:25:32.909599 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:32.909576 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4dd374d8-2dc8-45ff-9f9f-a82c019259af-metrics-certs\") pod \"maas-api-7fd9fcc7c9-dmvph\" (UID: \"4dd374d8-2dc8-45ff-9f9f-a82c019259af\") " pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-dmvph" Oct 31 20:25:32.919815 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:32.919787 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvzbq\" (UniqueName: \"kubernetes.io/projected/4dd374d8-2dc8-45ff-9f9f-a82c019259af-kube-api-access-pvzbq\") pod \"maas-api-7fd9fcc7c9-dmvph\" (UID: \"4dd374d8-2dc8-45ff-9f9f-a82c019259af\") " pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-dmvph" Oct 31 20:25:33.094649 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:33.094612 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-dmvph" Oct 31 20:25:33.376014 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:33.375989 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7fd9fcc7c9-dmvph"] Oct 31 20:25:33.378317 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:25:33.378289 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd374d8_2dc8_45ff_9f9f_a82c019259af.slice/crio-efd2101663abfb014f5e2ed007a87d3fa80f1897ccc24e46e046aceab61a9c1f WatchSource:0}: Error finding container efd2101663abfb014f5e2ed007a87d3fa80f1897ccc24e46e046aceab61a9c1f: Status 404 returned error can't find the container with id efd2101663abfb014f5e2ed007a87d3fa80f1897ccc24e46e046aceab61a9c1f Oct 31 20:25:33.379781 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:33.379761 4910 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 20:25:33.923597 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:33.923562 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-dmvph" event={"ID":"4dd374d8-2dc8-45ff-9f9f-a82c019259af","Type":"ContainerStarted","Data":"efd2101663abfb014f5e2ed007a87d3fa80f1897ccc24e46e046aceab61a9c1f"} Oct 31 20:25:34.931476 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:34.931440 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-dmvph" event={"ID":"4dd374d8-2dc8-45ff-9f9f-a82c019259af","Type":"ContainerStarted","Data":"aad72aad95cb76fc659924f305c614da260b93d157bca61c211869867ecd00bd"} Oct 31 20:25:34.931976 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:34.931614 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-dmvph" Oct 31 20:25:35.002297 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:35.002233 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-dmvph" podStartSLOduration=2.523687159 podStartE2EDuration="3.00221808s" totalImagesPullingTime="478.530921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:25:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:25:35.000201289 +0000 UTC m=+3969.363021681" watchObservedRunningTime="2026-10-31 20:25:35.00221808 +0000 UTC m=+3969.365038451" Oct 31 20:25:40.943628 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:25:40.943596 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7fd9fcc7c9-dmvph" Oct 31 20:25:42.712386 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:25:42.712349 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:26:12.712538 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:26:12.712455 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:26:42.715738 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:26:42.715701 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:27:12.711941 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:27:12.711884 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:27:42.711831 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:27:42.711788 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:28:12.711462 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:28:12.711424 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:28:32.028377 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:28:32.028340 4910 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7801e99-sdxbg"] Oct 31 20:28:32.031479 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:28:32.031450 4910 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7801e99-sdxbg"] Oct 31 20:28:33.753993 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:28:33.753960 4910 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48f18f27-a847-459a-ac41-64524cbf5df4" path="/var/lib/kubelet/pods/48f18f27-a847-459a-ac41-64524cbf5df4/volumes" Oct 31 20:28:42.711822 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:28:42.711778 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:29:12.716787 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:29:12.716703 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:29:13.332166 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:13.332132 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cvgtj_2dbfb4a1-52af-48dc-9739-938eff56a72d/manager/0.log" Oct 31 20:29:13.845759 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:13.845731 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lz987_35fff0bc-d871-4a3b-a6ab-0b6616c80dbd/server/0.log" Oct 31 20:29:15.009153 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:15.009118 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7fd9fcc7c9-dmvph_4dd374d8-2dc8-45ff-9f9f-a82c019259af/maas-api/0.log" Oct 31 20:29:18.286818 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:18.286781 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-pp8wj_0c073c51-9ff2-49db-a9f1-3e833a358623/manager/0.log" Oct 31 20:29:18.486558 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:18.486522 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-8xk49_2bfd6e75-7b11-4c98-aab3-8f3599cdca39/manager/0.log" Oct 31 20:29:18.677349 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:18.677322 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-cx5d8_b103a6dc-3af0-4ae2-b3fb-5c7b82042013/kuadrant-console-plugin/0.log" Oct 31 20:29:18.877328 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:18.877294 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-g55sk_29e3a181-1107-4d40-bc55-e093391fc65f/manager/0.log" Oct 31 20:29:19.296832 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:19.296787 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-q5rgw_d7afd221-6685-485c-933a-e3ac3b5e701b/manager/0.log" Oct 31 20:29:20.084682 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:20.084651 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8_23f65946-0acc-4386-89a8-3d350599d6ee/istio-proxy/0.log" Oct 31 20:29:21.390425 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:21.390389 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-f92ld_24c7b75b-240c-4c56-be8a-3159c9a57f05/istio-proxy/0.log" Oct 31 20:29:21.942984 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:21.942951 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-qgfcs_574d9e49-82cb-4a4e-ac28-ce3174ea17ab/istio-proxy/0.log" Oct 31 20:29:22.478597 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:22.478561 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-w7w5h_f5b43015-708e-4ef2-883e-cc121fe1c8e5/istio-proxy/1.log" Oct 31 20:29:22.679206 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:22.679172 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79_8d3f9166-9fbc-49b5-8292-90096c2f6c25/istio-proxy/0.log" Oct 31 20:29:23.219516 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:23.219484 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8c75d8d7f-tzzdl_70c870bc-c9e6-4d7e-bc99-ba556ce31cde/router/0.log" Oct 31 20:29:23.663344 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:23.663311 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-w7w5h_f5b43015-708e-4ef2-883e-cc121fe1c8e5/istio-proxy/0.log" Oct 31 20:29:27.039253 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:27.039224 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 20:29:27.101347 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:27.101320 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 20:29:32.685612 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:32.685576 4910 scope.go:122] "RemoveContainer" containerID="bf33f7e2beff4d84e16fa3a1103c6ac49f69561629391bd69b5314ea8c83ca60" Oct 31 20:29:36.049224 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:36.049192 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cvgtj_2dbfb4a1-52af-48dc-9739-938eff56a72d/manager/0.log" Oct 31 20:29:36.488624 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:36.488589 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-lz987_35fff0bc-d871-4a3b-a6ab-0b6616c80dbd/server/0.log" Oct 31 20:29:37.457429 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:37.457395 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7fd9fcc7c9-dmvph_4dd374d8-2dc8-45ff-9f9f-a82c019259af/maas-api/0.log" Oct 31 20:29:40.335387 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:40.335354 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-pp8wj_0c073c51-9ff2-49db-a9f1-3e833a358623/manager/0.log" Oct 31 20:29:40.531268 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:40.531225 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-8xk49_2bfd6e75-7b11-4c98-aab3-8f3599cdca39/manager/0.log" Oct 31 20:29:40.708321 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:40.708242 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-cx5d8_b103a6dc-3af0-4ae2-b3fb-5c7b82042013/kuadrant-console-plugin/0.log" Oct 31 20:29:40.893681 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:40.893648 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-g55sk_29e3a181-1107-4d40-bc55-e093391fc65f/manager/0.log" Oct 31 20:29:41.285536 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:41.285507 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-q5rgw_d7afd221-6685-485c-933a-e3ac3b5e701b/manager/0.log" Oct 31 20:29:41.847663 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:41.847631 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8_23f65946-0acc-4386-89a8-3d350599d6ee/istio-proxy/0.log" Oct 31 20:29:42.712156 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:29:42.712120 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:29:43.148010 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:43.147967 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-f92ld_24c7b75b-240c-4c56-be8a-3159c9a57f05/istio-proxy/0.log" Oct 31 20:29:43.699981 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:43.699951 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-qgfcs_574d9e49-82cb-4a4e-ac28-ce3174ea17ab/istio-proxy/0.log" Oct 31 20:29:44.254094 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:44.254060 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-w7w5h_f5b43015-708e-4ef2-883e-cc121fe1c8e5/istio-proxy/1.log" Oct 31 20:29:44.447452 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:44.447422 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79_8d3f9166-9fbc-49b5-8292-90096c2f6c25/istio-proxy/0.log" Oct 31 20:29:44.980977 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:29:44.980901 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8c75d8d7f-tzzdl_70c870bc-c9e6-4d7e-bc99-ba556ce31cde/router/0.log" Oct 31 20:30:12.711239 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:30:12.711200 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:30:32.706406 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:30:32.706372 4910 scope.go:122] "RemoveContainer" containerID="79f0283d38699e5e65ce98cc5e25be47dcdac237d91fbaed3c608929df9e7cc1" Oct 31 20:30:42.712818 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:30:42.712777 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:31:10.518281 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:10.518247 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lhtkb_9255a1d5-2514-441e-b942-6c39843e38f8/global-pull-secret-syncer/0.log" Oct 31 20:31:10.583412 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:10.583384 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-4mdb2_3a7f7d6d-958c-4973-9289-97f45526a431/connection-checker/0.log" Oct 31 20:31:10.617162 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:10.617138 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-vszkf_211efbea-29d0-4b2f-8c9e-6f3badcf77e5/connection-checker/0.log" Oct 31 20:31:10.650619 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:10.650593 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-xmfjb_1cc2220b-f30e-4879-b2cb-b2dd8e53b9f0/connection-checker/0.log" Oct 31 20:31:10.830947 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:10.830889 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wxdtf_1d0b685c-252c-459c-9dba-d19c1bc6e5d0/konnectivity-agent/0.log" Oct 31 20:31:10.930327 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:10.930288 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-116.us-east-2.compute.internal_956815bb47502a7ffa36d27c2bef7876/haproxy/0.log" Oct 31 20:31:12.711433 ip-10-0-142-116 kubenswrapper[4910]: E1031 20:31:12.711387 4910 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5nkwp" containerName="extractor" Oct 31 20:31:19.257243 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:19.257214 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-pp8wj_0c073c51-9ff2-49db-a9f1-3e833a358623/manager/0.log" Oct 31 20:31:19.294213 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:19.294186 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-8xk49_2bfd6e75-7b11-4c98-aab3-8f3599cdca39/manager/0.log" Oct 31 20:31:19.331163 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:19.331134 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-cx5d8_b103a6dc-3af0-4ae2-b3fb-5c7b82042013/kuadrant-console-plugin/0.log" Oct 31 20:31:19.396524 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:19.396493 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-g55sk_29e3a181-1107-4d40-bc55-e093391fc65f/manager/0.log" Oct 31 20:31:19.527292 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:19.527219 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-q5rgw_d7afd221-6685-485c-933a-e3ac3b5e701b/manager/0.log" Oct 31 20:31:20.599417 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:20.599382 4910 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hk24f/perf-node-gather-daemonset-66tq8"] Oct 31 20:31:20.602964 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:20.602945 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-66tq8" Oct 31 20:31:20.610116 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:20.609603 4910 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hk24f\"/\"default-dockercfg-pj4k7\"" Oct 31 20:31:20.614201 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:20.614178 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hk24f\"/\"openshift-service-ca.crt\"" Oct 31 20:31:20.614314 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:20.614263 4910 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hk24f\"/\"kube-root-ca.crt\"" Oct 31 20:31:20.614718 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:20.614700 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hk24f/perf-node-gather-daemonset-66tq8"] Oct 31 20:31:20.730474 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:20.730433 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bc6da757-a317-469f-8399-9d4eeb80129b-lib-modules\") pod \"perf-node-gather-daemonset-66tq8\" (UID: \"bc6da757-a317-469f-8399-9d4eeb80129b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-66tq8" Oct 31 20:31:20.730660 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:20.730505 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bc6da757-a317-469f-8399-9d4eeb80129b-podres\") pod \"perf-node-gather-daemonset-66tq8\" (UID: \"bc6da757-a317-469f-8399-9d4eeb80129b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-66tq8" Oct 31 20:31:20.730660 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:20.730560 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc6da757-a317-469f-8399-9d4eeb80129b-sys\") pod \"perf-node-gather-daemonset-66tq8\" (UID: \"bc6da757-a317-469f-8399-9d4eeb80129b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-66tq8" Oct 31 20:31:20.730660 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:20.730600 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8spt\" (UniqueName: \"kubernetes.io/projected/bc6da757-a317-469f-8399-9d4eeb80129b-kube-api-access-r8spt\") pod \"perf-node-gather-daemonset-66tq8\" (UID: \"bc6da757-a317-469f-8399-9d4eeb80129b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-66tq8" Oct 31 20:31:20.730660 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:20.730619 4910 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bc6da757-a317-469f-8399-9d4eeb80129b-proc\") pod \"perf-node-gather-daemonset-66tq8\" (UID: \"bc6da757-a317-469f-8399-9d4eeb80129b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-66tq8" Oct 31 20:31:20.831090 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:20.831024 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bc6da757-a317-469f-8399-9d4eeb80129b-lib-modules\") pod \"perf-node-gather-daemonset-66tq8\" (UID: \"bc6da757-a317-469f-8399-9d4eeb80129b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-66tq8" Oct 31 20:31:20.831090 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:20.831085 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bc6da757-a317-469f-8399-9d4eeb80129b-podres\") pod \"perf-node-gather-daemonset-66tq8\" (UID: \"bc6da757-a317-469f-8399-9d4eeb80129b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-66tq8" Oct 31 20:31:20.831458 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:20.831109 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc6da757-a317-469f-8399-9d4eeb80129b-sys\") pod \"perf-node-gather-daemonset-66tq8\" (UID: \"bc6da757-a317-469f-8399-9d4eeb80129b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-66tq8" Oct 31 20:31:20.831458 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:20.831152 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8spt\" (UniqueName: \"kubernetes.io/projected/bc6da757-a317-469f-8399-9d4eeb80129b-kube-api-access-r8spt\") pod \"perf-node-gather-daemonset-66tq8\" (UID: \"bc6da757-a317-469f-8399-9d4eeb80129b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-66tq8" Oct 31 20:31:20.831458 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:20.831168 4910 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bc6da757-a317-469f-8399-9d4eeb80129b-proc\") pod \"perf-node-gather-daemonset-66tq8\" (UID: \"bc6da757-a317-469f-8399-9d4eeb80129b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-66tq8" Oct 31 20:31:20.831458 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:20.831173 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bc6da757-a317-469f-8399-9d4eeb80129b-podres\") pod \"perf-node-gather-daemonset-66tq8\" (UID: \"bc6da757-a317-469f-8399-9d4eeb80129b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-66tq8" Oct 31 20:31:20.831458 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:20.831180 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bc6da757-a317-469f-8399-9d4eeb80129b-lib-modules\") pod \"perf-node-gather-daemonset-66tq8\" (UID: \"bc6da757-a317-469f-8399-9d4eeb80129b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-66tq8" Oct 31 20:31:20.831458 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:20.831281 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bc6da757-a317-469f-8399-9d4eeb80129b-proc\") pod \"perf-node-gather-daemonset-66tq8\" (UID: \"bc6da757-a317-469f-8399-9d4eeb80129b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-66tq8" Oct 31 20:31:20.831458 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:20.831456 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc6da757-a317-469f-8399-9d4eeb80129b-sys\") pod \"perf-node-gather-daemonset-66tq8\" (UID: \"bc6da757-a317-469f-8399-9d4eeb80129b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-66tq8" Oct 31 20:31:20.844347 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:20.844321 4910 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8spt\" (UniqueName: \"kubernetes.io/projected/bc6da757-a317-469f-8399-9d4eeb80129b-kube-api-access-r8spt\") pod \"perf-node-gather-daemonset-66tq8\" (UID: \"bc6da757-a317-469f-8399-9d4eeb80129b\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-66tq8" Oct 31 20:31:20.847103 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:20.847084 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-d4kds_4cd15118-0a59-4122-bdf1-56710c463262/cluster-monitoring-operator/0.log" Oct 31 20:31:20.916719 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:20.916649 4910 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-66tq8" Oct 31 20:31:21.066595 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:21.066557 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-74jl5_9543a93d-ce3e-4899-b961-68b45d88ff70/node-exporter/0.log" Oct 31 20:31:21.102361 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:21.102330 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-74jl5_9543a93d-ce3e-4899-b961-68b45d88ff70/kube-rbac-proxy/0.log" Oct 31 20:31:21.137187 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:21.137164 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-74jl5_9543a93d-ce3e-4899-b961-68b45d88ff70/init-textfile/0.log" Oct 31 20:31:21.194230 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:21.194205 4910 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hk24f/perf-node-gather-daemonset-66tq8"] Oct 31 20:31:21.195969 ip-10-0-142-116 kubenswrapper[4910]: W1031 20:31:21.195941 4910 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbc6da757_a317_469f_8399_9d4eeb80129b.slice/crio-5f46c0f65f2196c4888aad297beeb340a285b7afff77b6e507ab6fd1205e68f8 WatchSource:0}: Error finding container 5f46c0f65f2196c4888aad297beeb340a285b7afff77b6e507ab6fd1205e68f8: Status 404 returned error can't find the container with id 5f46c0f65f2196c4888aad297beeb340a285b7afff77b6e507ab6fd1205e68f8 Oct 31 20:31:21.755587 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:21.755549 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-66tq8" event={"ID":"bc6da757-a317-469f-8399-9d4eeb80129b","Type":"ContainerStarted","Data":"1c9fc13f75da9e748e15d3da2b152b6e7cfc9c410b9c98df1239d7b6561dab69"} Oct 31 20:31:21.756005 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:21.755689 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-66tq8" Oct 31 20:31:21.756005 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:21.755730 4910 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-66tq8" event={"ID":"bc6da757-a317-469f-8399-9d4eeb80129b","Type":"ContainerStarted","Data":"5f46c0f65f2196c4888aad297beeb340a285b7afff77b6e507ab6fd1205e68f8"} Oct 31 20:31:21.778255 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:21.778210 4910 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-66tq8" podStartSLOduration=1.7781983380000002 podStartE2EDuration="1.778198338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 20:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:31:21.777464364 +0000 UTC m=+4316.140284748" watchObservedRunningTime="2026-10-31 20:31:21.778198338 +0000 UTC m=+4316.141018710" Oct 31 20:31:23.339966 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:23.339904 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-28mzh_9cf7d7f5-3936-45b3-ba2c-597cf99cfb89/networking-console-plugin/0.log" Oct 31 20:31:23.920051 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:23.920013 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/2.log" Oct 31 20:31:23.925469 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:23.925440 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-6vpwc_76e07820-d888-4138-a41a-59ff4565accd/console-operator/3.log" Oct 31 20:31:25.264968 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:25.264931 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-xgrs8_ffb7495b-ef69-455c-99aa-22389d8c6d6e/volume-data-source-validator/0.log" Oct 31 20:31:26.543553 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:26.543527 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5nm2g_5418e7b8-1471-4aa5-ad70-5d60b87c572a/dns/0.log" Oct 31 20:31:26.572711 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:26.572680 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5nm2g_5418e7b8-1471-4aa5-ad70-5d60b87c572a/kube-rbac-proxy/0.log" Oct 31 20:31:26.773823 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:26.773798 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-85mq2_28c13795-a1de-4761-b3fc-feef76ac730d/dns-node-resolver/0.log" Oct 31 20:31:27.589684 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:27.589654 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8m482_84f5548f-7d70-4391-89c9-3bd3c1784717/node-ca/0.log" Oct 31 20:31:27.771022 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:27.770991 4910 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-66tq8" Oct 31 20:31:28.696954 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:28.696895 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8_23f65946-0acc-4386-89a8-3d350599d6ee/istio-proxy/0.log" Oct 31 20:31:30.217486 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:30.217435 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8_23f65946-0acc-4386-89a8-3d350599d6ee/istio-proxy/0.log" Oct 31 20:31:31.290350 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:31.290318 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-f92ld_24c7b75b-240c-4c56-be8a-3159c9a57f05/istio-proxy/0.log" Oct 31 20:31:31.407303 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:31.407265 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-qgfcs_574d9e49-82cb-4a4e-ac28-ce3174ea17ab/istio-proxy/0.log" Oct 31 20:31:31.525356 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:31.525323 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-w7w5h_f5b43015-708e-4ef2-883e-cc121fe1c8e5/istio-proxy/0.log" Oct 31 20:31:31.527353 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:31.527331 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-w7w5h_f5b43015-708e-4ef2-883e-cc121fe1c8e5/istio-proxy/1.log" Oct 31 20:31:31.559023 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:31.558990 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79_8d3f9166-9fbc-49b5-8292-90096c2f6c25/istio-proxy/0.log" Oct 31 20:31:31.664951 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:31.664905 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8c75d8d7f-tzzdl_70c870bc-c9e6-4d7e-bc99-ba556ce31cde/router/0.log" Oct 31 20:31:32.339208 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:32.339171 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-f92ld_24c7b75b-240c-4c56-be8a-3159c9a57f05/istio-proxy/0.log" Oct 31 20:31:32.400674 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:32.400649 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-qgfcs_574d9e49-82cb-4a4e-ac28-ce3174ea17ab/istio-proxy/0.log" Oct 31 20:31:32.460978 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:32.460941 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-w7w5h_f5b43015-708e-4ef2-883e-cc121fe1c8e5/istio-proxy/0.log" Oct 31 20:31:32.462449 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:32.462426 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-w7w5h_f5b43015-708e-4ef2-883e-cc121fe1c8e5/istio-proxy/1.log" Oct 31 20:31:32.480832 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:32.480809 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-5cf4665f7f-bxq79_8d3f9166-9fbc-49b5-8292-90096c2f6c25/istio-proxy/0.log" Oct 31 20:31:32.547681 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:32.547645 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8c75d8d7f-tzzdl_70c870bc-c9e6-4d7e-bc99-ba556ce31cde/router/0.log" Oct 31 20:31:34.465097 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:34.465066 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q8j58_92aaddbc-6268-443c-b3bd-9f74ab6b9a4b/serve-healthcheck-canary/0.log" Oct 31 20:31:35.141243 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:35.141212 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-mtmp6_599fc0e5-05d8-46b0-90b3-60818a371161/insights-operator/0.log" Oct 31 20:31:35.142121 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:35.142100 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-mtmp6_599fc0e5-05d8-46b0-90b3-60818a371161/insights-operator/1.log" Oct 31 20:31:35.170586 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:35.170552 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5nkwp_4e729724-d310-4884-a2d0-54af5308124c/kube-rbac-proxy/0.log" Oct 31 20:31:35.200457 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:35.200429 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5nkwp_4e729724-d310-4884-a2d0-54af5308124c/exporter/0.log" Oct 31 20:31:35.231852 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:35.231809 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5nkwp_4e729724-d310-4884-a2d0-54af5308124c/extractor/0.log" Oct 31 20:31:36.687215 ip-10-0-142-116 kubenswrapper[4910]: I1031 20:31:36.687173 4910 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-84d4d45d7cmgld8_23f65946-0acc-4386-89a8-3d350599d6ee/istio-proxy/0.log"